DPO drops the reward model and the PPO loop. It optimizes preference pairs directly with a closed-form loss, anchored to a frozen reference by a beta term.
Imagine teaching a chef which dish people prefer. The old way hires a food critic to score every plate, then runs a slow trial and error loop where the chef keeps cooking and the critic keeps grading. It works but it is fiddly and expensive. The new way skips the critic entirely. You hand the chef pairs of plates and say plainly: people liked this one more than that one. The chef adjusts directly from those comparisons. A gentle rule keeps the chef from drifting too far from their original style. Same goal, far fewer moving parts. That new way is DPO: learn from preference pairs directly, no separate scorer, no looping simulation, just a clean supervised-style nudge.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
DPO versus RLHF is one of the most common alignment questions in 2026 interviews, and most candidates get the headline right but the mechanism wrong. They know DPO is simpler. They cannot say precisely what it removes or why removing it is mathematically sound. The whole point of the question is to separate people who memorised a slogan from people who understand the reformulation.
Both methods solve the same problem: take a supervised-fine-tuned model and align it to human preferences expressed as comparisons between responses. The difference is entirely in the machinery. RLHF reaches the goal through a reward model and a reinforcement-learning loop. DPO reaches the same goal through a single supervised-style loss, having proven that the reward model and the loop are unnecessary in the offline setting.
The key insight to internalise is that DPO is not a lighter version of RLHF that swaps in a cheaper reward model. It is a reformulation that eliminates the reward model analytically and replaces the policy-gradient loop with one offline objective. Once you see the derivation, the operational differences (offline data, no sampling, implicit KL, stability) all follow.
The stakes are practical, not academic. A PPO run keeps four models live, depends on a delicate balance of clip range, value-loss coefficient, and KL coefficient, and can silently reward-hack a flawed reward model into producing confident garbage. A DPO run looks like a slightly unusual supervised job. For most teams that difference decides whether alignment ships in a week or stalls for a quarter, which is why the open-weight ecosystem swung hard toward DPO after 2023. This deep dive walks the RLHF pipeline, the DPO reparameterisation, the role of beta, the offline versus online trade-off, and exactly when each one is the right tool.
What classic RLHF actually runs
Classic RLHF is a three-stage pipeline, and naming all three stages is the first thing an interviewer listens for.
- SFT. Fine-tune the base model on high-quality demonstrations so it follows instructions at all. The result is the policy you will refine and, later, the reference you anchor to.
- Reward modelling. Collect human preference pairs and train a separate reward model to assign a scalar score, typically under a Bradley-Terry assumption that the probability of preferring one response is the sigmoid of the reward difference.
- PPO. Run reinforcement learning. The policy samples completions, the reward model scores them, and a clipped policy-gradient update maximises expected reward minus a KL penalty to the SFT reference.
The objective being optimised in stage three is reward minus a KL term:
This works and produced the first aligned chat models. But it keeps up to four models in memory at once: the policy, the reference, the reward model, and the PPO value head. It samples on-policy every step, and it is notoriously sensitive to reward hacking and hyperparameters.
Reward hacking deserves a sentence of its own, because it is the failure mode that haunts RLHF. The reward model is only a proxy for human judgement, trained on a finite set of comparisons. The policy is a relentless optimiser pointed straight at that proxy. Given enough steps it finds the gaps: responses that score high on the reward model but that humans would dislike, like sycophancy, padding, or formatting tricks. The explicit KL penalty to the reference is the main guardrail against this, which is one reason the KL term is not optional in either method.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | DPO | RLHF (PPO) |
|---|---|---|
| Reward model | None (implicit, reparameterised) | Separate trained reward model |
| Sampling | Offline, precollected pairs | On-policy, samples each step |
| Loss | Closed-form pairwise log-ratio | Clipped policy gradient on reward |
| KL to reference | Implicit via beta and frozen reference | Explicit penalty term |
| Stability and cost | Stable, cheap, simple to run | Fragile, costly, four models in memory |
| Quality ceiling | Capped by the preference set | Higher with good online reward |
Real products, models, and research that use this idea.
- Meta's Llama 4 post-training pipeline uses DPO-style preference optimization on top of SFT for instruction alignment.
- Hugging Face's TRL library ships DPOTrainer alongside PPOTrainer, and the Zephyr models popularised SFT then DPO as a recipe.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhere exactly does the reward model go in the DPO derivation?
Start from the KL-regularized RLHF objective, write its closed-form optimal policy, then solve for the reward in terms of the policy and reference log-ratio. Substituting into the Bradley-Terry preference model cancels the partition function.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying DPO is just RLHF with a smaller reward model. DPO has no reward model and no policy-gradient loop at all; it is a single offline loss over preference pairs.
60 second bullets to scan on the way to the call.
What DPO removes versus what RLHF keeps
The three stages of classic RLHF
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.