Unsupervised phase - pre-training on large corpus of unlabeled text = learning general-purpose representations via self-supervised learning - next token prediction.
Supervised alignment phase
To better align to end tasks and user preferences. Superficial Alignment Hypothesis (SAH) = most knowledge is learned during pre-training, while alignment refines the delivery format.
Supervised Fine-Tuning (SFT) teaches LLM to follow instructions by training it on many examples of accurate responses to instruction-based prompts. Good results require high-quality dataset.
Reinforcement Learning from Human Feedback (RLHF) - optimizes the model using human-provided feedback: multiple responses to the same prompts are generated and ranked by annotators => reward model is trained on this preference data with output = scalar reward is maximized (LLM is optimized) via the Proximal Policy Optimization (PPO) algo – SOTA for RL, special algo by OpenAI that utilize policy gradient methods - they search the space of policies rather than assigning values to state-action pairs. Rejection sampling is also used for RLHF (generates observations from a distribution).
Reward model in RLHF takes a prompt w/full chat history + response as input and predicts a preference score, usually it’s the same architecture and weights as the LLM, but its classification head (next token pred) is replaced with a regression head (preference estimation) and model is fine-tuned on preference data
Rejection sampling – LLM
PPO takes 1 sample from model per iteration (iterative updates after each sample). Rejection sampling takes K responses from LLM for each prompt, scores each w/reward model, and fine-tunes on best response (Llama 2 70B used for rejection sampling to train all other models).
Rejection sampling fine-tuning uses the same model (i.e., at the beginning of the RLHF round) to generate an entire dataset of high-reward samples that are used for fine-tuning in a similar manner to SFT (rejection sampling fine-tuning (RFT)).
For best performance, RFT includes best samples from all RLHF iterations, not just current one.
Rejection sampling for first four rounds of RLHF; final round = RFT + PPO sequentially (PPO last).
Proximal policy optimization (RL)
Trains agent to learn how to act in an env. in order to maximize reward (by OpenAI).
Adjusts policy (agent’s strategy) by gradient ascent to maximize the reward.
Clipped policy gradient objective - prevents large updates - more stable and reliable improvement.
Actor-Critic Method: 'actor' updates policy based on feedback from 'critic' which evaluates the policy.
Multiple Epochs per Update: runs through data multiple times (epochs) - better sample efficiency.
RLFH PPO requires 4 models in memory simultaneously: 1) Reward Model for human preferences: fitted & frozen, provides scalar reward, 2) Policy Model (Actor): being trained to maximize reward for predicting the token sequence, but making sure it is not too far from original reference model => 3) Reference Model: frozen SFT model used for KL divergence penalty to prevent reward hacking (when LLM learns to output garbage that is rewarded highly like emojis or backslashes) 4) Value func./model (aka Critic): Terminology disambig. - The value function V(s) is a math object: given a state (prompt + tokens generated so far), it estimates the expected total future reward, if generation continues under the current policy, to reduce variance during training => stabilize training. The value model (small regression head attached to the policy, trained jointly with it) is a neural net that implements the value function. Critic is just the actor-critic name for it.
PPO relies on "Advantage" - measures how better specific action (token) is vs. avg. expected action = difference betw. actual return and value predicted by Critic (in a simple case when lambda = 1 - close to reality). Return = discounted sum of all future rewards.
Reward model → produces scalar reward r (attached to final token, w/per-token KL penalties folded in).
Critic / value model → produces value V(s_t) at each token position.
GAE (Generalized Advantage Estimation) → estimator combining per-step rewards and Critic's values across time steps into advantage: it sums discounted temporal-difference (TD) residuals, δ_t = r_t + γV(s_{t+1}) − V(s_t), weighted by (γλ)^k. The λ parameter trades bias against variance: at λ=0 you get one-step TD advantage; at λ=1 it collapses to (total actual return − value predicted by Critic V(s_t)) - the above simple case.
Why this reduces variance: raw rewards = noisy training signal - decent completion to easy prompt and great completion to hard prompt can get similar scores. Subtracting expected reward (baseline) recenters signal around "better or worse than expected," shrinking variance of policy-gradient estimate w/out biasing it.
Two PPO methods: a) PPO-Clip (Clipped Surrogate Objective) - prevents new policy from deviating far from old one in a single update, clipping by ε [1-ε, 1+ε] (typically ε=0.2): de facto PPO standard due to its simplicity and strong empirical performance; b) PPO-KL - adds dynamic KL-divergence term to objective function to penalize large deviation of new policy from the old one (dynamic - if KL divergence too high, the coefficient increases to enforce stronger constraints; if too low, the coefficient decreases to allow larger updates). KL divergence - measure of how far two proba distributions are from each other.