TRANSFORMERS
Transformer contains a stack of Encoder layers (Encoders) and Decoder layers. The Encoder stack and the Decoder stack each have their corresponding Embedding layers. Output layer generates the final output.
Data inputs for both the Encoder and Decoder, which contains:
Embedding layer
Position Encoding layer
Each Encoder in the encoder stack contains:
Multi-Head Attention layer
Feed-forward layer
Each Decoder in the decoder stack contains:
Two Multi-Head Attention layers
Feed-forward layer
Output (top right) — generates the final output, and contains:
Linear layer
Softmax layer.
Input Representation in Transformers
Tokenization = breaking down input seq into smaller units – tokens (words, subwords or chars (punctuation)).
Embeddings map tokens to high-dimensional vectors capturing semantic and syntactic info about words.
Positional embeddings - capture position of each token within input seq (as Transformer processes input simultaneously, not sequentially as RNNs and LSTMs. In original Transformer paper, Pos encodings calculated w/sine and cosine f(x) are added to word embeddings => combined representation capturing both meaning and position.
Subword tokenization – splits rare words into smaller units to deal with complex words and misspellings, but keeping frequent words as unique entities (more efficient + managing the size).
Byte Pair Encoding (BPE) - used by OpenAI (GPT-2/3) and Meta (RoBERTa). Starts with characters, iteratively merges the most frequent pairs of adjacent symbols until desired vocabulary size. E.g. "lowest" → "low" + "est".
WordPiece - introduced by Google, used in BERT, DistilBERT. Starts with characters, iteratively merges based on maximizing likelihood (proba) of training corpus. Algo compares likelihood of training corpus tokenized with vs. without the merged token => the merge that gives the biggest likelihood increase wins. This merge typically has fewer tokens + more frequent subwords have higher probabilities. E.g., "playing" → "play" + "##ing". Disambiguation: SentencePiece = library that implements WordPiece, BPE, other tokenization algos.
Positional embeddings - vectors that encode the position of each token in seq because unlike RNNs / LSTMs transformers don’t understand the order of input tokens (crucial for NLU). Input embeddings represent tokens in a high-D space where similar tokens are closer together, but they don’t contain order info. Positional embeds + input embeds = transformer understands both meaning AND position of tokens.
Absolute Positional Embeddings - Encode exact position in seq (pos. 0, 1, 2, 3...) with a fixed representation regardless of context. Position 5 always gets the same encoding which doesn't depend on other pos. in seq. Limited generalization to longer unseen sequences.
1. Fixed (Sinusoidal) Positional Encodings
Not learned - formula-based: sine/cosine funcs of different frequencies to each position index.
Embeddings are continuous and generalize to unseen sequence lengths.
From original Transformer paper (Vaswani et al., 2017)
2. Learned Positional Embeddings
Learned during training as parameters. Each pos. has its own optimized embedding vector.
Limited to max sequence length during training.
Used in BERT, GPT-2/3, and many other models.
Relative Positional Embeddings - encode relative distance betw. positions: how far apart two tokens are (pairwise relations), e.g. encoding for pos. 5 attending to pos. 3 is based on the offset (-2). Better generalization to unseen sequence lengths. Used in Transformer-XL, T5 (relative bias), Llama, GPT-NeoX (RoPE).
Key Advantage of Relative (like RoPE) - translation invariance: relationship between token #5 & token #9 is same as between #105 & #108. More efficient for tasks where relative order matters more than absolute position.
Rotary Positional Embeddings (RoPE) - power long-context in most modern LLMs. Attention cares about relative distance, not absolute position. RoPE makes long-context generalization far more stable than learned position embeddings because it encodes relative offsets directly inside attention which is in line with how attention works => extrapolates better to longer sequences.
How RoPE works:
Queries and keys are split into pairs of dimensions - each pair = 2D vector.
These vectors are rotated by an angle based on token position.
Closer tokens have similar rotations
Rotation preserves vector magnitude but changes direction.
During Q·K attention, these phase differences encode relative distance naturally.
Residual connections - output of each sub-layer (MHA + FFN) is added to its input, and the sum is normalized (layer normalization) before being fed to the next sub-layer. Allows model to preserve info from earlier layers and addresses vanishing gradients and the fact that with greater network depth (more layers in Dl model), accuracy gets saturated and degrades rapidly. H(x) = F (x) + x, but network still trains by SGD w/backpropagation.
Transformer-XL (w/extra-long context) overcomes issues w/long-range dependencies (translation and language modeling) by implementing a recursive mechanism that connects different segments => allows to store and access info from prev. segments.
Vision Transformers (ViT) - image = sequence of patches. Image classification, competes with CNNs, achieves SOTA results on ImageNet.