DPO: what's the closed-form trick that lets it skip the reward model?

Classic **RLHF** has two stages. First you train a reward model on human preference pairs so it can score any answer. Then you run reinforcement learning, usually PPO, to push your language model toward high-reward answers, while a KL penalty keeps it from drifting too far from the original model. That pipeline is fragile. The reward model can be gamed, PPO is sample-hungry, and you have to keep several models in memory at once. **DPO** asks a sharper question: do we actually need the reward model as a separate object? It turns out there is a known formula for the best policy given a reward, and you can run that formula backwards. The reward can be rewritten as how much more likely your model makes an answer compared to the starting model. Once the reward is written that way, you substitute it into the preference formula and the reward function cancels out of the math. What remains is a simple loss on chosen versus rejected pairs. So DPO trains like ordinary supervised learning. Feed it pairs, make the chosen answer more likely than the rejected one, done. No reward model, no PPO loop, no online sampling.

**DPO** and RLHF share a starting point. Both assume a Bradley-Terry preference model, where one response beats another with probability sigmoid of their reward gap. RLHF fits that reward as an explicit model, then runs PPO against it with a KL leash to a reference policy. The trick is a known closed form. The KL-regularised reward-maximising policy is the reference distribution reweighted by the exponentiated reward. That relation can be inverted. So the reward equals beta times the log-ratio of policy to reference, plus a constant that depends only on the prompt. Now substitute that reward expression back into Bradley-Terry. The prompt-only constant cancels in the difference. The explicit reward function disappears. A classification loss on preference pairs is all that remains. It is: $$\mathcal{L}_{\text{DPO}} = -\log \sigma\!\left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right)$$ Here beta sets the KL strength. The reference policy is the frozen SFT checkpoint, run with no gradient. Beta sets how hard the policy is anchored to it. Small beta allows aggressive drift; large beta keeps it near the SFT model. Training is then one supervised pass on pairs, with no value function and no online rollouts.