PPO vs DPO, what's the practical difference?
PPO runs an RL loop against a trained reward model with a KL leash; DPO collapses that into one supervised loss over preference pairs, with no reward model and no RL.
Imagine teaching a chef to cook the way customers like. The PPO way: hire a food critic (a reward model) and train them up. Then put the chef in a kitchen where the critic scores every dish and the chef adjusts, dish after dish, in a feedback loop. It's powerful but expensive: you need the critic, the kitchen, and lots of iterations. The DPO way: skip the critic entirely. Show the chef pairs of dishes (here's a winner, here's a loser). Then use a single math trick that nudges the chef toward winners and away from losers without any feedback loop. Both end up with a chef who pleases customers. DPO is cheaper and simpler; PPO can sometimes get a slightly better chef if you're patient.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
PPO and DPO are the two dominant ways to align an LLM with human preferences after supervised fine-tuning. PPO is the RL algorithm in the InstructGPT recipe, which OpenAI also used for the first ChatGPT. DPO is the 2023 reformulation that turned the same problem into a supervised loss and now dominates the production landscape.
PPO needs a reward model and DPO doesn't.
The shared objective
Both methods optimize the same fundamental objective: maximize expected reward from a learned preference model, subject to a KL constraint that keeps the policy close to an SFT reference. The constraint matters; without it, the policy will reward-hack into degenerate outputs that score high but are unusable.
Formally, the objective is:
where r is the reward (learned from human preference data) and pi_ref is the SFT model. Beta controls the strength of the KL leash. A small beta gives the policy more freedom to chase reward, and a large beta keeps it tight to the reference.
The two methods differ in how they solve this. PPO solves it by trial: train a reward model, then use RL to climb the objective. DPO solves it analytically. The optimal policy has a closed form, which can be substituted back into the preference-pair likelihood to produce a single supervised loss.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Concern | PPO | DPO |
|---|---|---|
| Reward model needed | Yes, trained separately | No, derived analytically |
| Training loop | RL with rollouts + advantages | Supervised over preference pairs |
| Memory footprint | 4 models (policy, value, reference, reward) | 2 models (policy, reference) |
| Hyperparameter sensitivity | High (beta, clip, lr, KL target) | Moderate (mainly beta) |
| Wall clock cost | Days to weeks | Hours to days |
| Quality ceiling | Higher with tuning | Comparable in most cases |
Real products, models, and research that use this idea.
- InstructGPT and the original ChatGPT alignment used PPO, the canonical reference architecture for RLHF.
- Llama 3 and Llama 3.1's instruct variants used rejection sampling, SFT and DPO. Meta reported that DPO needed less compute than PPO at that scale. Meta describes Llama 4 post-training as lightweight SFT, then online RL, then lightweight DPO.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you debug a DPO run that's producing degenerate outputs?
Check KL from reference, plot beta sweep, inspect preference-pair quality. Degenerate outputs usually mean beta too low (policy ran away) or noisy pair labels.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Calling DPO 'just SFT on chosen responses'. DPO uses both chosen and rejected; the contrastive loss is the whole point.
60 second bullets to scan on the way to the call.
The pieces a PPO stack needs vs what DPO drops
The DPO loss form and what beta controls
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.