Machine Learning at Facebook

MACHINE LEARNING AT FACEBOOK

The Facebook Field Guide to Machine Learning

it’s important to understand how the entire development process works. It’s not only what happens during the training of your models, but everything that comes before and after, and how each step can either set you up for success.

1. Problem definition

The right set up of ML problem is often more important than the choice of algorithm. If not done properly, one can end up solving a wrong problem. Think into data if there are simple tasks, analyze different aspects of data. One can greatly simplify the process by defining simpler tasks that can act as good proxies for your ultimate goal. Who is the audience?

Summary:

1. Determine the right task for your projecting

2. Simple is better than complicated

3. Define labels and training example precisely. Edge cases?

4. Don’t prematurely optimize

2. Data

Hard to underestimate how preparing the quality training data is super-important. Building datasets is an active process – you must be sure that your data preparation processes prepare data in the RIGHT way.

Define training sample content, labels (1 or 0?)

Consider:

1. Data recency and real-time training (old data may have different structure). Sometimes need to get fresher data with time via real-time queries

2. Training / prediction consistency (online / offline)

3. Records and sampling – avoid missing rows / data. Look for changes in data distribution

3. Evaluation

• it’s important to have a clear plan for how to evaluate the performance of your model,

• Offline evaluation on logged data, online experimentation – on live traffic; may be more time-consuming, but gives better insights.

• Need a train / val / test set. If model performs well on train & eval, but not test => overfitting on training data

• Build a baseline model and see if the main model is better (if not – overfitting)

• Val – cross-validation, k chunks. Drawbacks – recent and non-recent results mixed. Progressive split is better when train on past data and predict on future data.

• Don’t throw away any useful data.

• Compute metrics that depend on task at hand. They should be: a) interpretable – compare with baseline, always must be improved), b) sensitive to improvements in the model.

• Results must be statistically significant, if not – increase the size of test data.

• Model calibration = sum pred / sum labels. If model is over-calibrated on test set, it does not generalize well on new data.

• Once offline evaluation is complete, test the candidate models on live traffic

Summary:

1. Evaluate offline before evaluating online

2. Evaluate both the choice of data and the kind of statisstics you calculate (both should reflect the online use case)

3. Don't evaluate and train on the same things

4. When evaluating, understand where the performance comes from

4. Features

ML cycle: Data => Features => Model => Evaluation => Data ...

Engineered feature - relevant to the output + interpretable by model. FB doesn't separate ML research from engineering implementation - researcher commit code to prod.

Choosing right feature for right model. Choice of features depends on:

• Model architecture

• Properties

• Special cases

• Amount of training data

Types of features:

• Categorical (k-out-of-n encoding),

• Continuous (repr. quantities) - powerful, but can decrease interpretability or represent a feedback loop.

• Derived features - output of other models

Feature breakage: the semantics of features should not change. Sometimes this can happen, but then it should be a new feature (don't just change old values) + train a model w/new features.

Avoid the difference in feature semantics during training and prediction time - must be same definitions / formulae.

Data/feature leakage - features give away the label (e.g. splitting the data by time stamp :). There is probably a leakage when the model performance is too good to be true

Feature coverage - what fraction of dataset has value for a feature. One can infer missing features, but maybe not when the coverage is < 10%. Coverage can change with time - monitor it.

5. Model

Picking a model

Linear models r interpretable, debuggable, fast in inference, good for large data. If little data and a few features - linear model. Challenge - most real-world problems are not linear.

Dealing with challenge - one can inject non-linearity into linear models, a) break down users by country and create new features, b) pretrain transformations in original features and use as features for linear model, e.g. use leaves of decision tree models in linear models, c) use a pretrained model, take the last layer for transfer learning. Careful, because now have to maintain more models and log more data.

Interpretability and ease of debugging

• Interpretability depends on the use case - do you need to explain model's results to others? E.g. why a post is banned

• Simpler models are easier to debug in the beginning. Once you know more about use case, more opaque models may be OK

• Understanding the theory is important - bias-variance tradeoff => if we break down the dataset and find a dimension when the model is not calibrated - model is biased there.

Data volume

Data volume - large dataset => DL. This is for more mature ML problems, but hard to debug. If good intuitiveness - more traditional ML.

Fine-tuning

• HPs (LR, regularization)

• Model architecture settings (feature interaction for linear m,, depth / leaves, # trees in tree-based models, # / type / width of layers in NNs

Considerations for fine-tuning:

• Efficiency, e.g. Jupyter notebooks allow to combine models, analysis, notes

• Transparency - name training sets in a meaningful way, otherwise it's not clear how results r achieved

• Reproducibility of experiments for new error analysis to see if the old experiments still hold

• Automation of HP tuning. Best loss may not be always for prod - infra costs

• Sometimes better gains come from inscreasing infra capacity, rather than better features or models

Comparing models

• Apples with apples - if trained on different data, still need the same test set!

• FB uses normalized entropy - how much better is our model's negative log-likelihood vs. model that makes a constant prediction. If > 1, smth is incorrect.

• Keep track of models trained and the data they were trained on

6. Experimentation

Making your experiments actionable.

Offline-online gap - difference between offline and online metrics and the effect of this on real world outcomes.

Gold standard for online evaluation - randomized A/B test. Users split into conttol and test group (the latter sees the new system).

1. Minimize time to first online experimental. Do it ASAP (2-year project, do it in 3 months). Measure the effect of code changes and the effect of ML model, for instance if you keep adding new components down the road in your offline tests. Back test: evaluate new model on 1% of online data (99% is processed by old model), then deploy the new model and evaluate it on 1% of online data on the old model. Be especially vigilant if results are neutral - A/A test?

2. Isolate engineering bugs from ML performance

3. Test models in presence of real-world feedback loops, when the old test data bocomes the new train data

4. Tips:

   • Be able to triangulate the cause of any changes - pinpoint the cause of a shift in any online metric

   • Measure the right things - measure everything, as many metrics as possible for each A/B test (FB uses hundreds), even optimizing only one metric. They all may be useful to validate the correctness of experiment

   • Have a backup plan - how to turn off the new functionality if things go south, extra kill switches in the code

   • Calibrate - how well avg preds match avg response rate: sum of preds / sum of realized events. Easy to monitor online and simple for sanity checking performance. If model miscalibrated on train set - not properly trained. If on test data - doesn't generalize well. If on online data - online-offline gap

Full "subtitles": https://medium.com/@nrkivar/facebook-field-guide-to-ml-3056900e7930