A team is starting reasoning post-training and must choose between DPO and GRPO. Describe when DPO is the better first move and when GRPO is preferable instead. Cover data requirements, exploration needs, and compute profile.
Start with DPO when you have strong offline preference pairs. Choose GRPO when you need online RLVR exploration with verifiable rewards and rollout sampling.
DPO is like studying from a textbook of good and bad worked examples, with no live practice games needed. GRPO is like playing many practice matches with instant scoreboards that tell you if your answer was right. Pick DPO when you already have labeled better/worse traces. Pick GRPO when the model must discover new strategies on hard math or code and you can check answers automatically.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
DPO versus GRPO is not a popularity contest. It is a data and compute topology question. Reasoning post-training projects fail when teams pick GRPO without verifiable rewards because DeepSeek did. They also fail when teams pick DPO because it is cheaper, when no labeled pairs exist for hard traces.
What DPO needs and delivers
DPO optimizes from static chosen vs rejected completion pairs under a preference objective. It needs no live rollouts during training steps. Data sources: human rankers, AI judges, or verifier-labeled traces (pass vs fail).
DPO wins when pairs are high quality and cover the behaviors you want. It is cheap relative to GRPO because you skip K× generation every step.
It loses when hard frontier behaviors never appear in the chosen set. DPO cannot invent unseen reasoning.
DPO pair quality beats quantity. A thousand verified chosen/rejected CoT pairs from a strong teacher beats a million noisy pairs from a weak base model.
Label economics: human rankers are expensive, and verifiers are cheap at scale. GRPO uses verifiers when pair data is scarce.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Teams polish distilled reasoning students with DPO on filtered teacher traces after GRPO teachers exist
- DeepSeek-R1 core discovery uses GRPO RLVR, not DPO-only training
What an interviewer would ask next. Try answering before peeking at the approach.
QWhen would you run GRPO then DPO sequentially?
GRPO discovers behaviors. DPO then stabilizes them on filtered winner/loser pairs from teacher rollouts. This is common when polishing distilled models.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Defaulting to GRPO for every project or claiming DPO replaces verifiable-reward exploration.
60 second bullets to scan on the way to the call.
State offline labeled pairs requirement for DPO
Note DPO cheaper without live rollouts
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.