L1 and L2 Regularization
Regularization - adding a penalty term to the cost function to discourage the weights from getting large. L2 or Ridge regularization - minimizes the squared magnitude of weights, shrinks all the coefficient by the same proportions, but eliminates none. |
 |
|---|---|
| L1 or Lasso penalty – minimizes absolute value of weights, can shrink to zero some coefficients (feature selection). |  |
Regressions summary
Linear Regression - More Details
Simplest - sum of squares of all coefficients => modified linear regr error function:
lamb
da = regularization parameter.
To minimize error - gradient descent algo = keep updating the model parameters to move closer to the values causing min. cost.
repeat until convergence ( with regularization):
With some manipulation the above equation can also be represented as:
First term (1 – alpha*lambda/m) always < 1 and reduces coeff by some amount on every update.
Logistic Regression
Cost function with regularization:
Where g() is the sigmoid (logistic) f(x):
Other Forms of Regularization
Explicit Regularization
These are techniques that add a direct penalty term to the loss function or explicitly constrain the model:
Elastic Net → Combines L1 (Lasso) and L2 (Ridge) penalties.
Group Lasso / Sparse Group Lasso → Promotes sparsity at the group or feature-block level.
Max-norm regularization → Constrains the norm of the weight vector to be below a fixed threshold.
Weight decay → Equivalent to L2 regularization but implemented directly in the optimization step.
Orthogonality regularization → Encourages weight matrices (especially in RNNs or CNNs) to be close to orthogonal, improving stability.
Spectral norm regularization → Constrains the largest singular value of a weight matrix, controlling Lipschitz constants.
Nuclear norm regularization → Promotes low-rank solutions by penalizing the sum of singular values.
Label smoothing → Regularizes classification by softening one-hot targets.
🔹 Implicit Regularization
These are methods that don’t add an explicit penalty, but the procedure itself biases the model toward simpler or more generalizable solutions:
Early stopping → Stops training before overfitting occurs, preventing weights from fitting noise.
Dropout (sometimes considered explicit) → Adds stochastic regularization by randomly zeroing activations during training.
Stochastic Gradient Descent (SGD) dynamics → SGD has an implicit bias toward flatter minima, which generalize better.
Batch normalization / Layer normalization → Normalize activations, implicitly regularizing the model by reducing internal covariate shift.
Data augmentation → Artificially enlarges the dataset (flips, rotations, mixup, CutMix, etc.), reducing overfitting.
Noisy training → Adding noise to inputs, hidden activations, or weights (Gaussian noise, dropout) improves robustness.
Smaller architectures / parameter sharing → CNN weight sharing, low-rank factorization, or pruning implicitly regularize by limiting capacity.
Ensembling / Bagging → Training multiple models and averaging predictions reduces variance.
Self-distillation / Knowledge distillation → Training with a teacher model’s soft predictions guides smoother decision boundaries.
Learning rate schedules → Large-to-small learning rates (cosine, step decay, 1-cycle) bias optimization toward simpler minima.
✅ Summary:
Explicit regularization: Adds direct penalties/constraints on weights (L1, L2, Elastic Net, spectral norm, label smoothing, etc.).
Implicit regularization: Emerges from the training process (SGD noise, early stopping, dropout, normalization, data augmentation, ensemble / bagging, small models, distillation).