GRPO (Group Relative Policy Optimization)
Used to train reasoning LLMs with RL. Simplifies PPO by removing Value (Critic) model (group-relative baseline instead); the reward model is dropped when rewards are verifiable (math/code checkers) — the standard reasoning-RL setup — otherwise a frozen reward model still scores each. GRPO samples multiple outputs for each prompt (group) and uses their mean reward within the group to estimate the Advantage = (each output’s reward - mean reward) / group std. Clipping & KL d. are kept too.
Reasoning models - are typically trained with RL variants that optimize for outcome-based rewards: models generate multiple reasoning chains/trajectories and are rewarded based on final answer correctness.
Direct Preference Optimization (DPO) - stable, simpler, computationally lightweight algo, higher performing than RLHF. RLHF - complex and unstable w/many hparams to tune + needs 4 models. DPO uses a single stage of policy training - learning policy directly from preference pairs w/out a reward model’s significant hparam tuning – applying a simple loss function optimized directly on a dataset of preferences {(x, yw, yl)} = {(prompt, preferred, dispreferred responses)} = classification problem on the human preference data. Pros: Needs 2 models instead of 4 for PPO (reference + fine-tuned). Cons: PPO performs better, but it’s more difficult to tune.
PPO vs. DPO: PPO advantages: stable, reliable, efficient, simple. Drawback: HP sensitivity, computationally demanding (4 models). DPO - only 2 models (easier), but suffers from out-of-distribution data. U don’t have to choose: Llama 3 training: pretraining -> SFT -> rejection sampling -> PPO -> DPO.
Identity Preference Optimization (IPO) - DPO tends to quickly overfit on preference dataset => IPO adds a regularization term to DPO loss to train models to convergence w/out early stopping.
Kahneman-Tversky Optimization (KTO) simplifies alignment & dispenses w/binary preferences, matches or surpasses DPO w/out relying on comparative preferences - defines a new HALO loss f(x) (Human-aware loss) using individual examples labelled as "good" or "bad" (thumbs up / down) => uses an adapted PPO for simplified offline, one-step alignment using existing preference dataset - no update of reference model to enhance stability. Thumbs up /down labels are much easier to acquire in practice - e.g. customer interaction data to align LLMs to desirable outcomes (e.g., sales made). KTO HF TRL implemented DPO, IPO, and KTO in DPOTrainer(). How much data - 10-100K examples for SFT and 50K for DPO.
Odds Ratio Preference Optimization (ORPO)
Fine-tunes and aligns LLMs (SFT + RLHF / DPO) in a single step - no reward or SFT models (ORPO - simpler than DPO and RLHF, performs on par w/DPO). Working with preference data, based on log odds ratio this method introduces a penalty to the NLL loss f(x) (negative log likelihood), to favor generations in the chosen response sets. ORPO shows faster training, lower memory, good results.
Simple Preference Optimization (SimPO)
New RLHF method improves simplicity and training stability, no reference model, similar to DPO, but outperforms DPO & ORPO on benchmarks + reduces time by ~20% & GPU memory by ~10% vs. DPO. Uses avg. log proba as reward & ensures larger gap between chosen and rejected responses. Built w/HF TRL.