When would you pick PPO based RLHF over DPO for a production alignment run?
Pick PPO when a reliable reward can score new samples, offline pairs miss what the policy writes now, and you can pay for four models and tuning. Otherwise DPO is simpler and cheaper.
Imagine training a new chef. The DPO way is to study a book of past dish comparisons from food critics. It is cheap and calm, but it covers only the dishes in the book, and other chefs cooked most of them. The PPO way is to cook new dishes every day and have a taster score each one. The chef learns from their own mistakes and can find better dishes. But it needs a good taster: if the taster loves salt, the chef learns to oversalt. It also needs a bigger kitchen staff, which costs more. Pick daily tasting when you trust the taster, the book does not cover your menu, and you can pay for the staff.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
PPO-based RLHF and DPO aim at the same objective: maximize a preference reward while staying close to a reference model. DPO solves it offline, on a fixed set of ranked pairs, with two networks and a simple loss. PPO solves it online. It samples new answers from the current policy, scores them with a reward model, and learns from those scores with four networks and a generation phase.
So the question is when online learning is worth its cost. PPO wins when three conditions hold together: a reward signal that stays reliable on new samples, a gap that fixed pairs cannot close, and the budget and skill to run an RL system. When any one of them is missing, DPO or a middle path is usually the better first choice.
What the online loop buys
DPO learns only from the answers in its dataset. Those answers are often off-policy: written by other models, or by an earlier version of the policy. The loss raises the chosen answer relative to the rejected one, but it never checks what the policy itself would write for the same prompt. Analysis of DPO shows it can even raise the probability of answers that appear in neither column of the data, because nothing in the loss penalizes them.
PPO closes that loop. Every step samples answers from the current policy, so the training signal always describes the model as it is now. If the policy starts a bad habit, the next batch of samples contains it, the reward model scores it down, and the update corrects it. This is exploration in a practical sense: the policy can try answers that are not in any dataset and keep the ones that score well.
That ability matters most when the best answers are rare in the existing data. Multi-step reasoning, math and code are the usual examples. A policy can find a correct solution path that no pair in the dataset shows, but only if something can score that path.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Concern | PPO RLHF | DPO |
|---|---|---|
| Core setup | Online RL loop with more components | Offline preference objective |
| Memory footprint | Usually higher | Usually lower |
| Iteration speed | Slower to tune | Faster to iterate |
| Best fit | Exploration-heavy regimes | Strong offline preference data |
Real products, models, and research that use this idea.
- Tulu 3 used DPO for its preference stage and then PPO for reinforcement learning with verifiable rewards on GSM8K, MATH and IFEval-style constraints.
- The Llama 3 report says its team explored PPO but chose DPO because DPO needed less compute for large models and performed better, especially on instruction following.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you check that a reward model is good enough to run PPO against?
Measure its accuracy on held-out preference pairs written by the current policy, and watch whether a separate judge agrees as the PPO reward rises.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Choosing PPO because it seems more powerful without checking that the reward model scores new samples well. With a weak reward model, PPO learns its flaws.
60 second bullets to scan on the way to the call.
What online, on-policy sampling gives PPO
Reward model quality as the precondition for PPO
Primary sources. Browse if you want the original framing.
- Xu et al., Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study
- Ivison et al., Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback
- Lambert et al., Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- Gao et al., Scaling Laws for Reward Model Overoptimization
- Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Same topic, related formats. Practice these next.