ACTIVE / WEAK / SEMI- / SELF-LEARNING
Active Learning
ML training paradigm aimed at reducing labeling costs - fewer labeled samples for similar accuracy. Instead of labeling the entire dataset, the model dynamically selects the most informative unlabeled samples to be annotated.
Cycle
Start with a small labeled dataset.
Train an initial model.
Select the most informative samples using a querying strategy (often based on uncertainty or diversity).
Annotate the selected samples and retrain the model.
Repeat until performance or labeling budget goals are met.
Unlike semi-supervised learning, active learning does not train directly on unlabeled data — it only decides which data should be labeled next.
Querying Strategies
Coreset sampling → Selecting a subset of training data that best represents the entire dataset, often used in active learning to reduce labeling costs.
Uncertainty-based: Pick samples where the model is least certain.
Least-Confidence → Selects samples where the model’s top prediction probability is lowest (least confident choice).
Entropy → Selects samples with the highest entropy across predicted probabilities (captures overall uncertainty across all classes).
Monte Carlo Dropout → Runs multiple stochastic forward passes with dropout enabled; selects samples with the largest variation (disagreement) in predictions, approximating Bayesian uncertainty.
Diversity-based: Ensure selected samples cover a broad range of feature space.
Clustering (e.g., k-means): Select samples closest to cluster centers to cover different regions of the feature space.
Core-set Selection: Choose points that best “cover” the dataset, minimizing distance between labeled and unlabeled sets.
Maximal Marginal Relevance (MMR): Balance relevance (informativeness) with diversity by penalizing similar samples.
Determinantal Point Processes (DPP): Probabilistically favor diverse subsets using determinants of similarity matrices.
Greedy k-Center (Farthest-First): Iteratively pick the sample farthest from already chosen ones, maximizing spread.
Performance-based: Target regions where the model performs poorly.
Hybrid/Bandit approaches: Combine multiple criteria into a reward-driven strategy.
Challenges
Uncertainty ≠ Usefulness: Low confidence may be caused by noise, outliers, or irrelevant data (e.g., Spanish tweets in an English dataset).
Messy data: Can pollute the labeled set if querying strategies rely only on confidence.
Computational cost: Iteratively retraining deep models can be expensive.
Deep learning
One rarely sees active learning in deep learning systems (though it’s still researched for niche cases like medical imaging, legal docs, or rare-event detection). Reasons:
DNNs require massive data anyway.
Retraining is costly.
Uncertainty estimation is unreliable.
Alternatives like self-supervised pretraining are more practical and powerful.
Difference from online learning
Online learning – training on streaming data which arrives little-by-little, and the model is updated sequentially. Training often at the point of collection (e.g, on the edge of a device).
Active learning – also sequential, but done offline when the dataset is usually fully available similar to supervised learning. Active learning selects small samples of data incrementally, and the model is either retrained on all records or updated with new data.
Other definitions
Margin sampling → An uncertainty-based active learning strategy that selects samples where the difference between the top two predicted class probabilities is smallest.
Query-by-committee → An active learning strategy where multiple models (a committee) vote on unlabeled samples; samples with the most disagreement are selected.
Budgeted active learning → Active learning under a fixed labeling budget, where the goal is to maximize model improvement per labeling cost.
Weak supervision
Weak Supervision is a paradigm of many approaches (heuristic labeling, distant supervision, programmatic labeling, etc.) that allow large datasets to be annotated cheaply and quickly without relying on full manual labeling – it trades rule-based or heuristic inputs for costly human annotations, combining them to produce large, labeled datasets efficiently (Snorkel is the most known implementation of this paradigm):
Experts or heuristics provide labeling rules (e.g., offensive post on LinkedIn = “all caps text,” + “mentions politics” + “post is about religion” + “post is negative” + “uses slang” + “topic is not professional”, etc.).
These rules act as noisy, imperfect labeling functions.
The system aggregates many such weak signals (e.g., weighting, majority vote, probabilistic model) to assign labels.
This enables the rapid annotation of millions of samples at low cost.
Beyond rules, weak supervision can also incorporate embeddings or other automated signals when explicit rules are hard to define (e.g., detecting objects in images).
Distant labeling = one form of weak supervision where labels are derived from raw data using an external knowledge source (knowledge base, ontology, dictionary, etc.) - automatically assigning noisy labels to raw data, enabling large-scale dataset creation with minimal human annotation.
How it works: if a knowledge base contains a relation “X is born in Y,” then any text mentioning “X” and “Y” together may be auto-labeled as a training example for the “is born in” relation.
Data programming → A weak supervision framework (popularized by Snorkel) where noisy labeling functions are combined into probabilistic labels for large datasets.
Labeling functions → User-defined heuristics, rules, or models that generate weak/noisy labels for data, used in weak supervision pipelines.
Semi-Supervised Learning
The model is trained on both labeled + unlabeled data – it uses a small labeled dataset to guide learning the structure in unlabeled data to achieve high accuracy with fewer labeled samples.
Self-training → A method where a model trained on labeled data generates pseudo-labels for unlabeled data, then retrains on both labeled and pseudo-labeled sets.
Tri-training → An extension of self-training where three models iteratively label data for each other, reducing noise compared to single-model pseudo-labeling.
Typical techniques:
Pseudo-labeling: Model predicts labels in unlabeled data → confident predictions are added to training.
Consistency regularization: Encourage the model to make stable predictions for the same unlabeled sample under perturbations (noise, augmentation).
Graph-based methods: Spread labels across a similarity graph built from all samples.
Example:
In text classification with only 1,000 labeled sentences but 100,000 unlabeled, the model learns from both by generating pseudo-labels for the unlabeled ones.
Label propagation → A semi-supervised algorithm that spreads label information from labeled to unlabeled samples through a similarity graph.
Co-training → A semi-supervised method where two models, trained on different “views” of the data, teach each other by labeling unlabeled samples.
Self-Supervised Learning
A paradigm where the model learns general-purpose representations from unlabeled data that can be transferred to downstream tasks with little labeled data – it uses the data’s internal structure to supervise itself, so no manual labels are needed
Typical techniques:
Masking: Predict masked words (BERT) or masked image patches (MAE - Masked Autoencoders).
Generative tasks: Reconstruct missing parts of input (autoencoders, GPT predicting next token).
Contrastive learning: the model learns representations by pulling augmented views of the same sample closer together in embedding space, while pushing different samples further apart.
SimCLR (Simple Framework for Contrastive Learning of Representations) - two augmented versions of same image should map to similar embeddings;
CLIP - images and text caption embeddings are aligned in one embedding space.
Example:
Pretrain BERT by predicting masked words in billions of unlabeled sentences → fine-tune on small labeled task for sentiment classification.Specific methods:
DINO (Self-Distillation with No Labels) → A self-supervised method where a student network learns from a teacher network’s augmented view embeddings without labeled data.
MAE (Masked Autoencoders) → A self-supervised vision method where parts of an image are masked, and the model learns by reconstructing the missing patches.
MoCo (Momentum Contrast) → A self-supervised contrastive learning approach using a momentum-updated encoder and a dynamic memory bank for negative samples.
Autoencoding pretext tasks → Self-supervised tasks where a model learns by reconstructing its input (e.g., denoising autoencoders, masked input reconstruction).
BYOL (Bootstrap Your Own Latent) → A self-supervised method where two networks predict each other’s representations of augmented views, without using negative samples.
Barlow Twins → A self-supervised approach that encourages embeddings of augmented views to be similar while reducing redundancy across feature dimensions.
SwAV (Swapping Assignments between Views) → A self-supervised method that uses clustering: it assigns cluster codes to different augmentations of the same image and trains the model to predict consistent cluster assignments.
Balancing dataset
ADASYN oversampling → A data balancing method that generates synthetic minority-class samples by focusing more on regions where the class is harder to learn.
SMOTE oversampling → A synthetic data generation technique that interpolates new minority-class samples between existing ones to balance class distribution.
Training Strategies
Self-paced learning → A training strategy where the model starts learning from easier examples and gradually incorporates harder ones, akin to curriculum learning
Curriculum learning → A training strategy where data is presented in increasing order of difficulty, mimicking human learning progression.