PPO vs DPO, what's the practical difference?

**PPO** (Proximal Policy Optimization) and **DPO** (Direct Preference Optimization) are two different recipes for aligning an LLM with human preferences. **PPO** is the older, classical recipe. It needs three pieces: a base model (after supervised fine-tuning), a separately trained reward model (which scores how good an output is, learned from human preference data), and an RL training loop. The loop has the model generate outputs, the reward model scores them, and the policy updates to maximize that reward while a KL-divergence penalty keeps it from drifting too far from the base. **DPO** is the newer, simpler recipe. It throws away the explicit reward model and the RL loop entirely. Instead, it uses preference pairs (here's a chosen output, here's a rejected output) directly in a supervised-style loss function that mathematically replicates what PPO would have done. The result is similar alignment with way less infrastructure.

Two different paths to the same destination, with very different operational footprints. **PPO** is the textbook RLHF stack: SFT model + reward model + RL loop with KL constraint. The reward model is trained separately on preference pairs to predict a scalar score. Then PPO optimizes the policy to maximize reward minus beta times KL-from-reference. You need rollouts, advantage estimation, clipped policy updates, and reward-model inference every step. It's memory hungry. It's hyperparameter sensitive. It's slow to converge. **DPO** collapses all of that into one loss: $$\mathcal{L}_{DPO} = -\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)}\right)$$ No reward model, no RL, no rollouts. Just a supervised loss over (chosen, rejected) preference pairs that's mathematically equivalent to PPO at the optimum. **Practical picture.** DPO is cheaper, faster, and easier to operate. PPO can edge out DPO on quality with careful tuning, especially on complex reward signals. The 2024-2026 default for most teams is DPO; PPO survives at frontier labs that have the infrastructure and need the last mile of quality.