BERT vs. GPT
Encoder-decoder architecture (T5, BART) – complex mapping from one text sequence to another for machine translation, summarization.
Decoder-only architecture (Generative Pretrained Transformers (GPT)). The model was trained using generative pre-training – it autocompletes the sequence by iteratively predicting the most probable next word (as a softmax distrib. over the entire vocab.). Text generation, Q&A. The computed representation for each token depends only on the left context = causal or autoregressive attention.
Autoregressive (causal) language model designed to handle sequential data with the attention mechanism, generative model that generates text given some input (iteratively predicts the next token as a softmax distrib. over the entire vocab.). Unsupervised pretraining + supervised fine-tuning for specific tasks + RLFH for dialog.
The decoder is trained autoregressively - at each step it predicts the next token given all previous tokens (probability distribution over the whole vocabulary). To make this work, the self-attention in the decoder is causally masked: each position can only attend to positions ≤ itself (no peeking at the future). Unidirectional text processing – consider left context to predict the next word.
Encoder-only architecture (Bidirectional Encoder Representations from Transformers (BERT) – converts input text into a rich numerical representation for further text classification, NER, Q&A, summary, etc. The computed representation for each token depends both on the left (before the token) and the right (after the token) contexts – bidirectional attention.
Bidirectional text processing - considers the entire sequence, both tokens to the left and to the right, when understanding context: more accurate representations for each word (bidirectional attention).
Pre-training strategies: masked language model (15% of random tokens are masked with [MASK] token - predicting the masked tokens based on their context) and next sentence prediction (predicting whether one sentence logically follows another sentence). In 50% of pairs a random (disconnected) sentence is chosen from the corpus instead of the subsequent sent. MLM and NSP are trained together to minimize the combined loss function.
BERT isn't generative - used for understanding the meaning of text: sentiment analysis, NER, Q&A.
BERT developers created two models:
BASE: # transformer blocks (L): 12 layers (one of the vertical layers on the diagram), Attention heads (A): 12 - exists inside every layer, within the Multi-Head Attention sub-layer of each Transformer block - the model splits its "attention" horizontally into multiple heads that work in parallel: one head focuses on grammar, another on context (what "it" refers to), etc., Hidden Units (H): 768 - total "width" of data, must be divided among the heads - 768 hidden units across 12 heads = each head processes a vector of size 768 / 12 = 64.
LARGE: # transformer blocks (L): 24 layers, Attention heads (A): 16, 1024 hidden units or size (H)
By having 12 or 16 heads, the model can "understand" 12 or 16 different types of relationships simultaneously
For comparison: GPT-3 has 175 B params, 96 layers. GPT-4 1.8T params, 120 layers.
Fine-tuning BERT - adding small layer to the core model:
Classification head (e.g. sentiment analysis) - similarly to Next Sentence classification = add a classification layer on top of Transformer output for [CLS].
Q&A tasks (e.g. SQuAD v1.1) - model receives a question re a text sequence and marks the answer in the sequence. Training Q&A model - two extra vectors marking the beginning and end of answer.
In Named Entity Recognition (NER) - model receives a text sequence and marks various types of entities (Person, Organization, Date, etc). Training NER model - feeding output vector of each token into classification layer predicting NER label.
 |
 |
Both decoder LMs and MLMs generate proba distrib. over entire vocabulary at each position. Difference:
In decoder inference you use the prediction at the last position to extend the sequence (autoregression), and
In MLM training, you use the predictions at the masked positions to reconstruct missing tokens - the training loss is computed only at the masked positions => BERT learns to “fill in the blanks,” not to generate text step by step.
MLM is usually part of encoder-only models, but it can also be adapted in encoder–decoder setups (like T5, BART) into denoising or span-masking objectives for pretraining. Decoder-only models normally don’t use MLM — they use causal LM (next token prediction).
Statistical Model (ChatGPT) + Symbolical Logic (Wolfram Alpha). WA - computational knowledge engine developed by Wolfram Research: answers factual queries by computing answers from externally sourced data.
BERT Varieties
BERT
Uses bidirectional self-attention with masked language modeling (MLM) and next-sentence prediction (NSP).
Token embeddings mix word content + position in a single vector.
RoBERTa - improves training
Keeps the same architecture as BERT, but improves training strategy:
Trained on much more data
Dynamic masking (masks change every epoch)
Removes NSP (found unnecessary)
Larger batches and longer training
Gains come from optimization, not new model design.
DeBERTa - improves architecture
Introduces architectural changes:
Disentangled attention: separates token content and positional information
Relative position modeling directly inside attention
This allows the model to reason more precisely about who attends to whom and where.
DistilBERT: a compressed BERT trained via knowledge distillation to be smaller and faster with minimal accuracy loss.
Fewer layers (e.g., 6 instead of 12), no NSP
How it’s trained
Uses knowledge distillation: learns to mimic a full BERT (“teacher”)
Optimized to match teacher outputs and hidden representations
Why it exists
~40% fewer parameters
~60% faster inference
~95–97% of BERT’s performance