7Forward the policy on (prompt + chosen) to get `logπ_θ(chosen)`
**DPO**, Direct Preference Optimization, trains a policy to prefer one response over another using a frozen reference model as a baseline. One training step has a fixed shape.
First, sample a batch. Each example is three things: a prompt, the chosen (preferred) response, and the rejected response.
Next, run four forward passes. Two come from the policy you are training: one on the prompt plus chosen, one on the prompt plus rejected. Two come from the reference model, which is the policy frozen at the start: same two sequences, no gradients.
Those four forwards produce four log-probabilities: `logπ_θ(c)`, `logπ_θ(r)`, `logπ_ref(c)`, `logπ_ref(r)`.
Now form the log-ratio difference. Take the chosen gap, `logπ_θ(c) minus logπ_ref(c)`, then subtract the rejected gap, `logπ_θ(r) minus logπ_ref(r)`. That single scalar measures whether the policy has moved toward chosen relative to rejected.
Finally, compute the loss `−log σ(β · difference)`, where β is a temperature, and backprop. Gradients flow into the policy only. The reference stays untouched.
DPO replaces RLHF's reward model and PPO loop with a single supervised-style step driven by preference pairs.
The step starts by sampling a batch of `(prompt, chosen, rejected)` triples. Then it runs **exactly four forward passes**: the policy on `(prompt + chosen)` and `(prompt + rejected)`, then the frozen reference on the same two sequences. Reference forwards run inside `torch.no_grad()` because no gradient should flow through them.
From those four passes you extract four sequence-level log-probabilities, summing token log-probs over the response tokens only (the prompt is masked out). Call them `logπ_θ(c)`, `logπ_θ(r)`, `logπ_ref(c)`, `logπ_ref(r)`.
Form the **log-ratio difference**: `[logπ_θ(c) − logπ_ref(c)] − [logπ_θ(r) − logπ_ref(r)]`. The chosen gap minus the rejected gap. Positive means the policy has moved toward preferring chosen over rejected relative to the reference.
Apply the DPO loss:
$$\mathcal{L} = -\log \sigma\!\left(\beta \cdot [\log\pi_\theta(c) - \log\pi_{\text{ref}}(c) - \log\pi_\theta(r) + \log\pi_{\text{ref}}(r)]\right)$$
β is a temperature, typically around 0.1, that controls how aggressively the policy diverges from the reference. Backprop the loss; gradients update only the policy.
Quick reference
TL;DR
Sample a (prompt, chosen, rejected) triple, run four forwards (two policy, two frozen reference), form the log-ratio gap on chosen minus the same gap on rejected, then backprop the sigmoid loss into the policy only.
Imagine grading two essays a student wrote on the same prompt, one good, one bad. You also keep last week's version of the student to use as a reference baseline. You ask both students to score each essay and write down four scores. Then you check: did the new student get more excited about the good essay relative to the old student than they did about the bad essay relative to the old student? If yes, great, keep nudging in that direction. If not, push harder so the gap grows. DPO is just that comparison made math: four scores, one preference signal, gentle updates only to the new student so they do not forget who they were.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You only have scalar reward signals rather than pairwise preferences DPO is built on (chosen, rejected) pairs; with scalar rewards, classical RLHF with PPO or KTO with signed utilities fits the data shape.
Memory is tight and you cannot hold a second model copy alongside the policy DPO doubles activation memory at minimum; SFT or a LoRA-only DPO variant is needed when the device cannot fit both forward graphs.
Preference data is extremely noisy or contradictory DPO assumes the Bradley-Terry preference model; high label noise causes the sigmoid loss to saturate without producing useful signal.
You need fine-grained control over the KL constraint that PPO exposes explicitly DPO's KL anchor is implicit through β and the reference; if you need a hard KL bound or adaptive KL, PPO-RLHF still gives more control.
Quick scan · 2–4 min
Everything important, quickly.
5 min: four forwards (two policy, two frozen reference) + sequence log-prob sum + log-ratio gap difference + sigmoid loss with β + policy-only backprop + memory trade-off.
Side-by-side
Step
Model
Input
Gradient?
1
n/a
Sample (prompt, chosen, rejected) triple
n/a
2
Policy θ
prompt + chosen
yes
3
Policy θ
prompt + rejected
yes
4
Reference (frozen)
prompt + chosen
no
5
Reference (frozen)
prompt + rejected
no
6
n/a
Form log-ratio gap difference
n/a
7
Policy θ only
Backprop sigmoid loss
yes
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face TRL's `DPOTrainer` is the canonical 2026 implementation; it runs the four forwards, masks the prompt, and supports sharing the reference across DP ranks.
2Llama 4 and DeepSeek V4 chat variants use DPO or its IPO / KTO descendants as the preference-alignment stage after SFT.
3Axolotl and LLaMA-Factory expose DPO recipes that load a reference model alongside the policy and stream four-forward batches.
4Zephyr 7B (HuggingFaceH4) was an early high-profile demonstration that DPO on UltraFeedback could match PPO-RLHF quality with simpler infra.
5Anthropic and Mistral report internal preference-alignment pipelines that ablate DPO, IPO, and KTO on top of their SFT checkpoints, all sharing the four-forward shape.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat does β actually control, and what happens at very large or very small β?
Sign in or go Pro to see the approach for this follow-up.
QWhy does DPO need no explicit reward model where PPO-RLHF does?
Sign in or go Pro to see the approach for this follow-up.
QHow do IPO and KTO modify the DPO loss, and what failure modes do they target?
Sign in or go Pro to see the approach for this follow-up.
QHow can you reduce the 2x memory cost of holding the reference alongside the policy?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Forgetting that the reference model needs both a chosen and a rejected forward, not one shared call. Without both reference log-probs you cannot form the gap difference DPO is built on.