Describe the three stages of the classical RLHF pipeline (as in InstructGPT / Ouyang et al. 2022). Then explain exactly where SFT and DPO sit, and how DPO simplifies the picture.
Classical RLHF is three stages: SFT, then a reward model on preferences, then PPO against it with a KL penalty. SFT comes first in both; DPO folds the last two into one preference loss.
Imagine training a new chef. First you show the chef many good dishes and say 'cook like this': that is supervised fine-tuning, the chef learns to follow instructions. Then you set two of the chef's dishes side by side and a taster says which is better, over and over; from those judgements you build a scorecard that predicts what tasters like. Finally the chef keeps cooking, you score each dish with the card, and the chef adjusts to score higher, while a rule stops the chef drifting into weird food nobody asked for. That last part is the reinforcement-learning step. DPO is a shortcut: skip building the scorecard and the slow trial loop, and learn straight from the 'this dish beats that dish' pairs in one pass.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5 min: name the three classical stages in order, explain why SFT must come first, describe reward model data and the PPO KL penalty, then show how DPO collapses stages 2 and 3 while keeping SFT as the reference.
| Stage | Classical RLHF | DPO pipeline |
|---|---|---|
| Stage 1 | SFT on demonstrations | SFT on demonstrations (same) |
| Stage 2 | Train a reward model on preference pairs | Folded into the DPO loss |
| Stage 3 | PPO against reward model with KL penalty | Folded into the DPO loss |
| Explicit reward model | Yes, a separate scalar-head model | No, the reward is implicit |
| Online sampling loop | Yes, PPO samples on-policy each step | No, offline on fixed preference pairs |
| Reference policy | SFT checkpoint anchors the KL penalty | SFT checkpoint is the reference in the log-ratio |
Real products, models, and research that use this idea.
- InstructGPT in 2022 established the canonical three-stage recipe: SFT, reward model, then PPO with a KL penalty against the SFT reference.
- Llama 4 post-training in 2026 runs SFT followed by preference optimisation, with DPO as the cheap default before any heavier online RL pass.
- DeepSeek V4 alignment combines an SFT stage with rule-based and preference-based RL, showing that SFT remains the shared prerequisite across recipes.
- Hugging Face TRL ships SFTTrainer, RewardTrainer, PPOTrainer, and DPOTrainer, mirroring exactly where each stage sits in the pipeline.
- Zephyr and many open-weight chat models popularised the SFT-then-DPO recipe, skipping a separate reward model and PPO entirely.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the reward model usually initialised from the SFT checkpoint rather than the raw base model?
QWhat exactly does the KL penalty in PPO regularise against, and what breaks without it?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Saying DPO replaces SFT. It does not. SFT is stage 1 in both pipelines, and the SFT checkpoint is the reference policy DPO trains against.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.