KNN
Classifies a new data point based on the majority class of its nearest neighbors - uses feature similarity => the new point is assigned a value based on how closely it resembles the points in the training set.
Non-parametric, instance-based learning, lazy learning algo: recommendation systems, anomaly detection, pattern recognition, and clustering.
KL Divergence (Kullback-Leibler Divergence) is a measure of how one probability distribution differs from another reference distribution. It quantifies the “information loss” when using Q to approximate P, or equivalently how much extra surprise is incurred if data generated by Pis instead modeled as Q. Formally asymmetric and always ≥ 0 (equals zero only when the two distributions match exactly).
Support Vector Machines
Searching for a hyperplane that maximizes the margin between the classes by mapping data to a high dimensional space. Regressor - same, but margin of tolerance (epsilon) is set in approximation to the SVM.
Take data to a multidim. space and separate it there (circular category) - kernel transformation to turn linear problems into non-linear (basis function for each point in a multidim. space - SVM algo uses a kernel trick: fast way to build this space w/out doing all calculations).
Key property of SVMs: only a small subset of training points (“support vectors” that lie on or near the decision boundary) determine the final classification boundary. Once you have found the points closest to the margin, only they matter for constructing the separating hyperplane - points farther away do not change it. Because of this:
The decision boundary is driven by the support vectors, rather than every point in the dataset.
Better scalability to high-dimensional data: Algos struggle in high-dim. spaces because they rely on all points. SVMs focus only on critical points near the margin.
Efficient kernel trick: when using kernels (e.g., polynomial, RBF, etc.), the model effectively separates data in a higher-dimensional feature space (possibly infinite-dimensional), but still only requires computation involving the support vectors => SVMs - computationally feasible even when the data dimensions grow large.
Random Forest
 |
Random forests are an example of an ensemble learner built on decision trees.
Binary splitting - each node in the tree splits the data into two groups using a cutoff value within one of the features. But danger of capturing noise in data - overfitting. Multiple overfitting estimators can be combined to reduce the overfitting. Bagging uses ensemble of such parallel strong estimators to average the results. Ensemble of randomized decision trees - random forest. |
Boosting
Ensemble of "weak" classifiers trained consecutively where misclassified data points are given higher weight at next steps, but the overall prob distribution is still 1 => lower overall bias. The bias prediction error is reduced by focusing on poor predictions and trying to model them better in the next iterations.
Bootstrapping
Statistical procedure of resampling with replacement a single dataset to create many simulated samples. This process allows you to calculate standard errors, construct confidence intervals, and perform hypothesis testing for numerous types of sample statistics.
Bagging (bootstrap aggregation)
Combination of "strong" learners with some vote trained on partial training data each using resampling with replacement => reduces variance.