Same topic, related formats. Practice these next.
DPO drops the reward model and the PPO loop. It optimizes preference pairs directly with a closed-form loss, anchored to a frozen reference by a beta term.
Imagine teaching a chef which dish people prefer. The old way hires a food critic to score every plate, then runs a slow trial and error loop where the chef keeps cooking and the critic keeps grading. It works but it is fiddly and expensive. The new way skips the critic entirely. You hand the chef pairs of plates and say plainly: people liked this one more than that one. The chef adjusts directly from those comparisons. A gentle rule keeps the chef from drifting too far from their original style. Same goal, far fewer moving parts. That new way is DPO: learn from preference pairs directly, no separate scorer, no looping simulation, just a clean supervised-style nudge.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5 min: what RLHF needs (reward model + PPO loop) + the DPO closed-form reparameterisation + the implicit KL via beta and reference + offline versus on-policy trade-offs + when PPO still wins + the DPO variants.
| Aspect | DPO | RLHF (PPO) |
|---|---|---|
| Reward model | None (implicit, reparameterised) | Separate trained reward model |
| Sampling | Offline, precollected pairs | On-policy, samples each step |
| Loss | Closed-form pairwise log-ratio | Clipped policy gradient on reward |
| KL to reference | Implicit via beta and frozen reference | Explicit penalty term |
| Stability and cost | Stable, cheap, simple to run | Fragile, costly, four models in memory |
| Quality ceiling | Capped by the preference set | Higher with good online reward |
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Saying DPO is just RLHF with a smaller reward model. DPO has no reward model and no policy-gradient loop at all; it is a single offline loss over preference pairs.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.