- 1Forward the policy on (prompt + rejected) to get `logπ_θ(rejected)`
- 2Sample a batch of (prompt, chosen_response, rejected_response) triples
- 3Forward the frozen reference on (prompt + rejected) to get `logπ_ref(rejected)` (no grad)
- 4Compute the DPO loss `−log σ(β · log-ratio difference)` and backprop into the policy only
- 5Forward the frozen reference on (prompt + chosen) to get `logπ_ref(chosen)` (no grad)
- 6Form the log-ratio difference: `[logπ_θ(c) − logπ_ref(c)] − [logπ_θ(r) − logπ_ref(r)]`
- 7Forward the policy on (prompt + chosen) to get `logπ_θ(chosen)`
Sample a (prompt, chosen, rejected) triple, run four forwards (two policy, two frozen reference), form the log-ratio gap on chosen minus the same gap on rejected, then backprop the sigmoid loss into the policy only.
Imagine grading two essays a student wrote on the same prompt, one good, one bad. You also keep last week's version of the student to use as a reference baseline. You ask both students to score each essay and write down four scores. Then you check: did the new student get more excited about the good essay relative to the old student than they did about the bad essay relative to the old student? If yes, great, keep nudging in that direction. If not, push harder so the gap grows. DPO is just that comparison made math: four scores, one preference signal, gentle updates only to the new student so they do not forget who they were.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Direct Preference Optimization (DPO) is the 2023 algorithm that replaced most of the moving parts of RLHF. Where PPO-based RLHF needs an explicit reward model, a value head, advantage estimation, and a separate KL constraint, DPO collapses all of that into a single supervised-style loss on preference pairs. The training step looks almost like SFT, except you run four forwards per example instead of one and the loss compares policy log-probabilities against a frozen reference.
The question is testing whether you can sequence those four forwards correctly and connect them to the closed-form loss. Get the count wrong and you either miss a log-probability the loss needs, or you spend gradient on the reference and break the algorithm.
The deep dive below walks through the step in seven stages, then explains the derivation that makes the four-forward shape inevitable rather than arbitrary.
Stage 1: sampling the preference triple
DPO consumes preference-labeled data. Each training example is a triple: a prompt x, a chosen response y_c, and a rejected response y_r. The chosen / rejected labels come from human annotators or, increasingly, from a strong model's pairwise judgments. UltraFeedback is the canonical 2024 dataset of this shape and remains a common starting point in 2026.
The batch is then a list of such triples. Unlike SFT, you cannot just train on (x, y_c) pairs and ignore y_r; the rejected response is what makes the loss informative. The two responses must come from the same prompt so that the comparison is conditional on identical context. Cross-prompt comparisons are not meaningful under the Bradley-Terry assumption DPO is built on.
A practical note: response lengths can differ between chosen and rejected, sometimes substantially. The implementation has to pad both to a common length per batch (or use packing) so the two policy forwards have compatible shapes. Length mismatches that bias the loss toward shorter or longer responses are a known DPO failure mode that the IPO and SimPO variants address.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Step | Model | Input | Gradient? |
|---|---|---|---|
| 1 | n/a | Sample (prompt, chosen, rejected) triple | n/a |
| 2 | Policy θ | prompt + chosen | yes |
| 3 | Policy θ | prompt + rejected | yes |
| 4 | Reference (frozen) | prompt + chosen | no |
| 5 | Reference (frozen) | prompt + rejected | no |
| 6 | n/a | Form log-ratio gap difference | n/a |
| 7 | Policy θ only | Backprop sigmoid loss | yes |
Real products, models, and research that use this idea.
- Hugging Face TRL's `DPOTrainer` is the canonical 2026 implementation; it runs the four forwards, masks the prompt, and supports sharing the reference across DP ranks.
- Llama 4 and DeepSeek V4 chat variants use DPO or its IPO / KTO descendants as the preference-alignment stage after SFT.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat does β actually control, and what happens at very large or very small β?
Large β pushes hard on each preference but lets the policy drift far from the reference, risking mode collapse. Small β keeps the policy close to reference but learns the preference slowly. β is the implicit KL coefficient.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forgetting that the reference model needs both a chosen and a rejected forward, not one shared call. Without both reference log-probs you cannot form the gap difference DPO is built on.
60 second bullets to scan on the way to the call.
The exact count and shape of forwards per DPO step
Which forwards carry gradient and which run under no_grad
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.