Sequence the forward passes and loss steps inside a single DPO training step

**DPO**, Direct Preference Optimization, trains a policy to prefer one response over another using a frozen reference model as a baseline. One training step has a fixed shape. First, sample a batch. Each example is three things: a prompt, the chosen (preferred) response, and the rejected response. Next, run four forward passes. Two come from the policy you are training: one on the prompt plus chosen, one on the prompt plus rejected. Two come from the reference model, which is the policy frozen at the start: same two sequences, no gradients. Those four forwards produce four log-probabilities: `logπ_θ(c)`, `logπ_θ(r)`, `logπ_ref(c)`, `logπ_ref(r)`. Now form the log-ratio difference. Take the chosen gap, `logπ_θ(c) minus logπ_ref(c)`, then subtract the rejected gap, `logπ_θ(r) minus logπ_ref(r)`. That single scalar measures whether the policy has moved toward chosen relative to rejected. Finally, compute the loss `−log σ(β · difference)`, where β is a temperature, and backprop. Gradients flow into the policy only. The reference stays untouched.

DPO replaces RLHF's reward model and PPO loop with a single supervised-style step driven by preference pairs. The step starts by sampling a batch of `(prompt, chosen, rejected)` triples. Then it runs **exactly four forward passes**: the policy on `(prompt + chosen)` and `(prompt + rejected)`, then the frozen reference on the same two sequences. Reference forwards run inside `torch.no_grad()` because no gradient should flow through them. From those four passes you extract four sequence-level log-probabilities, summing token log-probs over the response tokens only (the prompt is masked out). Call them `logπ_θ(c)`, `logπ_θ(r)`, `logπ_ref(c)`, `logπ_ref(r)`. Form the **log-ratio difference**: `[logπ_θ(c) − logπ_ref(c)] − [logπ_θ(r) − logπ_ref(r)]`. The chosen gap minus the rejected gap. Positive means the policy has moved toward preferring chosen over rejected relative to the reference. Apply the DPO loss: $$\mathcal{L} = -\log \sigma\!\left(\beta \cdot [\log\pi_\theta(c) - \log\pi_{\text{ref}}(c) - \log\pi_\theta(r) + \log\pi_{\text{ref}}(r)]\right)$$ β is a temperature, typically around 0.1, that controls how aggressively the policy diverges from the reference. Backprop the loss; gradients update only the policy.