OPTIMIZATION
Adjusting model’s parameters to minimize a loss function:
define how weights and biases are updated during training to converge to a (local / global) minimum of the loss f(x) + to find a balance between fitting training data and generalizing to unseen data.
Deep neural networks involve millions or even billions of parameters, creating highly non-convex loss surfaces full of valleys and saddle points. Optimization tackles this by introducing adaptive LRs, momentum, curvature approximations, and regularization. Transformers, CNNs, or large-scale recommender systems would be impossible to train efficiently without optimizers.
TYPES OF OPTIMIZATION METHODS
🔹 First-Order Methods
Stochastic Gradient Descent (SGD): Updates parameters using minibatches of data; the foundation of deep learning optimization.
SGD w/Momentum: momentum - adds a velocity term to smooth gradient updates & speed up convergence. You accumulate speed in directions where gradients consistently point the same way — like a ball rolling downhill.
SGD w/Nesterov Momentum (Acceleration) - improves momentum by taking a look ahead step before computing the gradient. This provides a corrective term — so the optimizer compensates for a potential overshoot.
Coordinate Descent: Updates one parameter (or coordinate) at a time, useful for sparse problems.
Proximal Methods: Handle constraints or regularization terms by combining gradient steps with proximal operators (used in convex optimization).
🔹 Adaptive Gradient Methods - dynamically adjust LR per parameter, improving convergence.
AdaGrad: Adapts learning rate per parameter, scaling down updates for frequently updated parameters.
AdaDelta: Improves AdaGrad by reducing aggressive learning rate decay.
RMSProp: Maintains a moving average of squared gradients to normalize updates.
Adam: Combines momentum (first moment) and adaptive scaling (second moment); the most widely used optimizer.
AdamW: Adam with proper weight decay regularization (decoupled).
Adamax: Adam variant based on the infinity norm.
AMSGrad: Adam variant that ensures non-increasing learning rates to stabilize convergence.
Nadam: Adam with Nesterov momentum.
RAdam: Rectified Adam, stabilizes training in the early stages by correcting variance.
Yogi: Variant of Adam with better control over variance of adaptive learning rates.
NovoGrad: Scales gradient updates by global gradient statistics, more memory-efficient than Adam.
Adafactor: Memory-efficient optimizer (reduces storage of second-moment estimates), often used in large NLP models.
LAMB: Layer-wise adaptive large-batch training optimizer (used in BERT, GPT training).
Lion Optimizer: Recent optimizer replacing momentum with a sign-based update rule, highly efficient.
Sophia Optimizer: Scales updates by a stochastic approximation of the Hessian diagonal; faster convergence in LLMs.
🔹 Second-Order & Curvature-Based Methods
Newton’s Method: Uses full Hessian matrix of second-order partial derivatives of the loss function for curvature (to make smarter updates); impractical for large deep networks.
Quasi-Newton Methods (L-BFGS): Approximate the Hessian with low-rank updates.
AdaHessian: Extends Adam by using approximate curvature information from the Hessian.
Shampoo Optimizer: Approximates second-order statistics using matrix factorizations.
K-FAC (Kronecker-Factored Approximate Curvature): Efficient approximation of the Fisher Information Matrix for large-scale models.
PRACTICAL ENHANCEMENTS IN OPTIMIZATION
🔹 Learning Rate Schedules
Control how aggressively updates are made across epochs
Step Decay: Reduces learning rate at fixed intervals.
Exponential Decay: Continuously decays learning rate by a constant factor.
Cosine Annealing: Smoothly decreases learning rate following a cosine curve.
1Cycle Policy: Cycles learning rate up and then down for rapid convergence.
Warmup Schedules: Gradually increase learning rate in early epochs to stabilize training.
🔹 LR Warm-up
Meaning: Start training w/small LR, gradually increase it to target LR over few thousand steps.
Why:
Prevents instability or divergence early on (in pre-trained models, large LR can cause catastrophic forgetting or overshooting past the optimum).
Allows the optimizer’s momentum to stabilize (Adam) before full-speed training.
🔹 LR Re-Decay (for continual pretraining)
Meaning: After warm-up + plateau period, gradually decrease (decay) LR again (cosine, exponential, or linear schedule).
Why:
As training continues, smaller updates help fine-tune the model more precisely.
In continual pre-training, you’re adapting an existing model, not training from scratch — so decay prevents overwriting prev. learned info and reduces forgetting.
Re-decay - decay happens again, after the model already underwent decay in earlier pre-training.
🔹 Training Stabilization Techniques
Gradient Clipping: Caps gradients to prevent exploding values.
Adaptive Gradient Clipping (AGC): Scales gradient clipping relative to parameter norms (also prevents exploding values).
Gradient Accumulation: Accumulates gradients across minibatches to simulate larger batch sizes on memory-constrained hardware.
🔹 Hybrid & Advanced Optimizers
Apollo Optimizer: Combines curvature information with adaptive step sizes for robustness.
Lookahead Optimizer: combines fast, noisy steps with slower, more stable updates, improving generalization.
Distributed and memory-efficient methods enable large-scale training of billion-parameter models:
Sharded Optimizers: Partition optimizer states across devices for distributed training efficiency.
ZeRO Optimizers: (Zero Redundancy Optimizer) splits optimizer states, gradients, and parameters across devices, enabling trillion-parameter training.
PagedAdamW: Memory-efficient AdamW variant designed for large-scale models.