Model Interpretability

Transformer interpret library for BERT-like models:

Attributing the prediction of DNN to input features.
Word attributions = average attention given by model to every token/subtoken from all the model’s layers. Default method - Integrated Gradients (or SHAP).
Positive attributions - tokens making the model predict the specific label.
Negative attributions - making the model predict other labels.

Transformer interpret is based on Captum library

From the video (see References)

Easy to extend this library to add perturbation and other algos. The library already uses SHAP and other algos in addition to Integrated Gradients

If we want to get attributions for input features with respect to target 0:

using the Integrated Gradients algo (code on left)
algo performs forward and backward passes and computes the integral of the gradients for the output target 0 with respect to inputs along the path from 0 to input
attributions have the same shape and dimensionality as inputs
magnitude of attributions – strength of importance signal (color-coded in green)

It is also possible to attribute target 0 to a certain layer, for example the second linear layer. All steps are similar, but the attributions are for the layer

From the blog (see References)

References (Interpretability)

1. Main article: Axiomatic Attribution for Deep Networks

2. Video with explanations: https://www.youtube.com/watch?v=iVSIFm0UN9I&ab_channel=PyTorch

3. Great blog: https://jalammar.github.io/explaining-transformers/