DPO drops both the separate reward model and the PPO loop, yet (under its assumptions) hits the same optimum as RLHF. Walk through the closed form trick that makes this possible. What role does β play, and what's the reference policy?
DPO inverts the KL-regularised RL optimum to write the reward as a log-ratio of policy to reference, then substitutes it into Bradley-Terry, leaving one classification loss on preference pairs.
Imagine grading essays by first training a judge, then coaching a writer to please that judge. That two-stage dance is RLHF: train a reward model, then run reinforcement learning against it. DPO notices something clever. There is a known formula linking the best possible writer to the judge's scores, so you can run the formula backwards. The writer's own behaviour reveals what the judge's scores must have been. Once you express the score that way, you plug it straight into the rule for which essay people preferred, and the judge cancels out of the math entirely. Now you just nudge the writer to make preferred essays more likely than rejected ones, using a simple comparison on pairs you already labelled. No separate judge, no fiddly reinforcement loop, just one steady training pass.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Direct Preference Optimization is the answer to a frustration that defined the first wave of RLHF: aligning a model meant juggling several models and a reinforcement learning loop that was notoriously hard to stabilise. You trained a reward model on human preference pairs, then ran PPO to push your policy toward high reward while a KL penalty kept it from collapsing onto the reward model's blind spots. Each stage had its own failure modes, and the whole thing was expensive to run and tune.
DPO's contribution is not a new objective invented from scratch. It is a re-derivation. It starts from exactly the same mathematical setup as RLHF, then shows that the reward model is an unnecessary intermediate. The reward is implicit in the policy itself, and once you see that, the entire pipeline collapses into a single supervised-style loss on the preference pairs you already have.
This deep dive walks the derivation end to end: the shared Bradley-Terry assumption, the closed-form solution to KL-regularised reward maximisation, the inversion that reads the reward off a log-ratio, the substitution that cancels the reward model, and the role of beta and the reference policy. It closes with the honest caveats, because at staff level the interesting question is not what DPO does but when it falls short of on-policy methods.
The shared setup: Bradley-Terry preferences
Both RLHF and DPO model human preferences with the Bradley-Terry model. Given a prompt and two responses, a labelled pair marks one as chosen and one as rejected. The model says the probability that the chosen response beats the rejected one is the sigmoid of the difference of their rewards.
Formally, with a latent reward function r over prompt-response pairs, the preference probability is the logistic of the reward gap:
The reward here is a real-valued score. In classical RLHF you fit it with a separate neural network, training that network by maximum likelihood on the preference data. That fitted reward model is what PPO later optimises against. The crucial thing to hold onto is that this preference probability depends only on the difference of rewards, never on their absolute scale. Any constant added to every reward leaves the sigmoid untouched, because the difference erases it. That property is exactly what lets DPO cancel an awkward term later.
It is worth being explicit about why both methods share this assumption. Humans are far better at comparing two responses than at assigning an absolute numeric score, so almost all preference data arrives as pairs. Bradley-Terry is the standard probabilistic model for turning such pairwise comparisons into a latent score, and it is the bridge that lets a discrete preference become a differentiable loss. RLHF and DPO disagree only on what to do with that bridge, not on the bridge itself.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | RLHF (reward model + PPO) | DPO |
|---|---|---|
| Reward model | Explicit network trained first | Implicit, the log-ratio of policy to reference |
| Optimisation | Online PPO with sampled rollouts | Offline supervised-style loss on pairs |
| Models in memory | Policy, reference, reward, often a critic | Policy plus frozen reference only |
| Stability and cost | Sample-hungry, tuning-heavy | Cheaper, more stable, easier to run |
| Exploration | Can sample new responses on-policy | Limited to the fixed preference dataset |
Real products, models, and research that use this idea.
- Hugging Face TRL ships a DPOTrainer used to align open models like Llama 4 and Mistral variants on community preference datasets.
- Zephyr-7B-beta was a landmark public model post-trained with DPO on UltraFeedback, showing DPO could rival PPO-aligned chat models.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the partition function in the closed-form optimum not appear in the DPO loss?
The partition term depends only on the prompt, not the response. Bradley-Terry uses the reward difference between two responses to the same prompt, so the term cancels exactly. Show it explicitly in the subtraction.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying DPO has no reward at all. It carries an implicit reward, the log-ratio of policy to reference, that is never fit as a separate network but still drives the whole loss.
60 second bullets to scan on the way to the call.
Bradley-Terry preference probability as a sigmoid of a reward gap
Closed form of the KL-regularised reward-maximising policy
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.