Bias–Variance, Ensembles & Model Drift

Bias Variance Trade Off

https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff

Central problem in supervised ML - choose a model that both accurately captures the regularities in training data AND generalizes well to unseen data (usually impossible).

Models with a lower bias in parameter estimation have a higher variance of the parameter estimates across samples, and vice versa – conflict of trying to simultaneously minimize two types of error:

• Bias (too simple models) - erroneous assumptions during learning – missing relations between features and target var (underfitting).

• Variance (too complex models) - sensitivity to small fluctuations in training data, modeling random noise (overfitting).

How it looks (underfitting and overfitting):

How to select best model:

Underfitting: the model has too few features and hence not able to learn from the data very well. This model has high bias. Consider the problem of predicting y from x ∈ R. The leftmost figure below shows the result of fitting a line to a data-set. Since the data doesn’t lie in a straight line, so fit is not very good (left side figure).

To increase model capacity, we add another feature by adding term x² to it. This produces a better fit ( middle figure). But if we keep on doing so ( x⁵, 5th order polynomial, figure on the right side), we may be able to better fit the data but will not generalize well for new data. The first figure represents under-fitting and the last figure represents over-fitting.

Conclusion – increase number of features to fight underfitting, but it can become overfitting if there are too many features

Approaches to reduce

There are three main options to address the issue of overfitting:

1. Reduce the number of features (feature selection): Manually select which features to keep. Doing so, we may miss some important information, if we throw away some features.

2. Regularization: Keep all the features, but reduce the magnitude of weights W. Regularization works well when we have a lot of slightly useful feature.

3. Early stopping: When we are training a learning algorithm iteratively such as using gradient descent, we can measure how well each iteration of the model performs. Up to a certain number of iterations, each iteration improves the model. After that point, however, the model’s ability to generalize can weaken as it begins to overfit the training data.

Other methods:

• Dimensionality reduction - simplify models

• Larger training set reduces overfitting (variance)

• Tunable parameters:

  • Linear and Generalized linear models – regularization.

  • DL – overfitting (variance) increases with # hidden units (arguable).

  • K-nearest neighbor model – low k means overfitting (variance).

  • In instance-based learning, regularization can be achieved varying the mixture of prototypes and exemplars. (instance-based – think kNN, when training data is in memory, and we compare new data points to it. Similar to exemplars in memory in psychology and prototypes – the most representative concepts for a category, e.g. couch for furniture)

  • Decision trees – greater depth = more overfitting (variance). Pruning controls overfitting (variance).

• Use mixture models and ensemble learning: a) boosting to lower bias (ensemble of "weak" classifiers trained consecutively where misclassified data points are given higher weight at next steps, but the overall prob distribution is still 1), and b) bagging or bootstrap aggregation to reduce variance (combination of "strong" learners with same vote trained on partial data each using resampling with replacement).

• Model validation methods (cross-validation) can optimize the trade-off

• Feature generation: polynomial features (x, x^2, x^3)

More recipes for DL (notes from Andrew Ng by someone):

• If your algorithm has a high bias:

  • Try to make your NN bigger (size of hidden units, number of layers)

  • Try a different model that is suitable for your data.

  • Try to run it longer.

  • Different (advanced) optimization algorithms.

• If your algorithm has a high variance:

  • More data.

  • Try regularization.

  • Try a different model that is suitable for your data.

• You should try the previous two points until you have a low bias and low variance.

• In the older days before deep learning, there was a "Bias/variance tradeoff". But because now you have more options/tools for solving the bias and variance problem, it’s really helpful to use deep learning.

• Training a bigger neural network never hurts

ML Aspects To Refresh

Boosting

Ensemble of "weak" classifiers trained consecutively where misclassified data points are given higher weight at next steps, but the overall prob distribution is still 1 => lower overall bias. The bias prediction error is reduced by focusing on poor predictions and trying to model them better in the next iterations.

Bootstrapping

Statistical procedure of resampling with replacement a single dataset to create many simulated samples. This process allows you to calculate standard errors, construct confidence intervals, and perform hypothesis testing for numerous types of sample statistics

Bagging (bootstrap aggregation)

Combination of "strong" learners with some vote each trained on partial training data using resampling with replacement) => reduces variance

N-fold cross-validation

Repeating train / val split n times and providing average metrics

Stratified cross-validation

Variation of N-fold that preserves the percentage of samples for each class

Model validation

Type 1: using the training data - analyzing the goodness of fit of the model (how predictions fit the actual data, main methods: chi-square, Kolmogorov-Smirnov, Anderson-Darling tests) or analyzing whether the residuals are random (e.g. regression model). Less adequate.

Type 2: not using the training data – does the model's predictive performance deteriorate non-negligibly when applied to new unseen data.

Model drift, model decay - degradation of ML model performance over time: lower accuracy. SEE ML SYSTEM DESIGN FOR MORE DETAILED ANSWER. Two types:

CONCEPT DRIFT

Relationship changed between input (independent) and target (dependent) variables. This means that the definition of what we are trying to predict changes so that our algorithm provides inaccurate predictions. This change can be gradual, sudden, or recurring.

• Gradual: fraudsters perfect their methods => historical fraudulent transaction data would not reflect a new strategy as fraud.

• Sudden: sharp increase in buying home fitness equipment increased by 18% vs. decrease on transportation services by 23% in 2020 (COVID).

• Recurring: seasonality - retail sales increase during Christmas.

DATA DRIFT

Figure 1. Changing age distribution can cause data drift.

INPUT DRIFT - statistical properties of input data change: the age distribution of users changes over time. Since the usage habits of young and old people are not the same, a model trained on young people’s usage data would provide inaccurate predictions for old people’s behavior.

OUTPUT DRIFT - label freq. changes.

How to deal with model drift?

Monitor performance: specialized model monitoring tools and MLOps platforms.

Check data quality

Some rapid performance changes can be due to problems in training data quality such as biases in data rather than concept or data drift. Also, if model is trained on high-quality data, but real-world data is lower in quality. Ignoring a known seasonality is not a good practice.

Retrain the model

Retrain with more recent data:

• Use only recent data if old data has become outdated,

• Use all available data if the old data wouldn’t cause inaccurate model predictions,

• Assign higher weights to recent data so that the model pays less attention to old data.

Another option is online learning where the model continuously learns in real-time with the data feed. This will enable the model to keep itself up to date with evolving datasets.

Rebuild the model

Different features, hyperparameters, model architectures, etc.

Changes in the digital environment can also lead to model decay (remember version of Numpy because of the version of pandas etc.) -