- 1Forward the policy on (prompt + rejected) to get logπ_θ(rejected)
- 2Sample a batch of (prompt, chosen_response, rejected_response) triples
- 3Forward the frozen reference on (prompt + rejected) to get logπ_ref(rejected) (no grad)
- 4Compute the DPO loss −log σ(β · log-ratio difference) and backprop into the policy only
- 5Forward the frozen reference on (prompt + chosen) to get logπ_ref(chosen) (no grad)
- 6Form the log-ratio difference: [logπ_θ(c) − logπ_ref(c)] − [logπ_θ(r) − logπ_ref(r)]
- 7Forward the policy on (prompt + chosen) to get logπ_θ(chosen)
Sample a (prompt, chosen, rejected) triple, run four forwards (two policy, two frozen reference), form the log-ratio gap on chosen minus the same gap on rejected, then backprop the sigmoid loss into the policy only.
Imagine grading two essays a student wrote on the same prompt, one good, one bad. You also keep last week's version of the student to use as a reference baseline. You ask both students to score each essay and write down four scores. Then you check: did the new student get more excited about the good essay relative to the old student than they did about the bad essay relative to the old student? If yes, great, keep nudging in that direction. If not, push harder so the gap grows. DPO is that comparison made math: four scores, one preference signal, gentle updates only to the new student so they do not forget who they were.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Direct Preference Optimization (DPO) is the 2023 algorithm that replaced most of the moving parts of RLHF. PPO-based RLHF needs an explicit reward model, a value head, advantage estimation, and a separate KL constraint. DPO collapses all of that into a single supervised-style loss on preference pairs. The training step looks almost like SFT, except for two things. You run four forwards per example instead of one and the loss compares policy log-probabilities against a frozen reference.
The four forwards must be sequenced correctly and connected to the closed-form loss. Get the count wrong and you either miss a log-probability the loss needs, or you spend gradient on the reference and break the algorithm.
Stage 1: sampling the preference triple
DPO consumes preference-labeled data. Each training example is a triple: a prompt x, a chosen response y_c, and a rejected response y_r. The chosen / rejected labels come from human annotators or, increasingly, from a strong model's pairwise judgments. UltraFeedback (released October 2023) is a standard dataset of this shape and remains a common starting point.
The batch is then a list of such triples. Unlike SFT, you cannot just train on (x, y_c) pairs and ignore y_r; the rejected response is what makes the loss informative. The two responses must come from the same prompt so that the comparison is conditional on identical context. Cross-prompt comparisons are not meaningful under the Bradley-Terry assumption DPO is built on.
Response lengths can differ between chosen and rejected, sometimes substantially. The implementation has to pad both to a common length per batch (or use packing) so the two policy forwards have compatible shapes. Separately, DPO is known to drift toward longer responses. SimPO targets this length bias by averaging log-probs over the response length. IPO targets a different failure: overfitting when preferences are close to deterministic.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Step | Model | Input | Gradient? |
|---|---|---|---|
| 1 | n/a | Sample (prompt, chosen, rejected) triple | n/a |
| 2 | Policy θ | prompt + chosen | yes |
| 3 | Policy θ | prompt + rejected | yes |
| 4 | Reference (frozen) | prompt + chosen | no |
| 5 | Reference (frozen) | prompt + rejected | no |
| 6 | n/a | Form log-ratio gap difference | n/a |
| 7 | Policy θ only | Backprop sigmoid loss | yes |
Real products, models, and research that use this idea.
- Hugging Face TRL's DPOTrainer is a widely used implementation. It runs the four forwards and masks the prompt. It can also precompute reference log-probs or, with a PEFT policy, use the base model with adapters disabled as the reference.
- Many open-weight chat models add DPO or a variant such as IPO, KTO or SimPO as a preference stage after SFT.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat does β actually control, and what happens at very large or very small β?
Large β anchors the policy tightly to the reference: the loss saturates after a small shift, so preferences move less. Small β lets the policy drift far from the reference, which risks over-optimizing the preference data and mode collapse. β is the implicit KL coefficient.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forgetting that the reference model needs both a chosen and a rejected forward, not one shared call. Without both reference log-probs you cannot form the gap difference DPO is built on.
60 second bullets to scan on the way to the call.
The exact count and shape of forwards per DPO step
Which forwards carry gradient and which run under no_grad
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.