Describe the three stages of the classical RLHF pipeline (as in InstructGPT / Ouyang et al. 2022). Then explain exactly where SFT and DPO sit, and how DPO simplifies the picture.
Classical RLHF is three stages: SFT, then a reward model on preferences, then PPO against it with a KL penalty. SFT comes first in both; DPO folds the last two into one preference loss.
Imagine training a new chef. First you show the chef many good dishes and say 'cook like this': that is supervised fine-tuning, the chef learns to follow instructions. Then you set two of the chef's dishes side by side and a taster says which is better, over and over; from those judgements you build a scorecard that predicts what tasters like. Finally the chef keeps cooking, you score each dish with the card, and the chef adjusts to score higher, while a rule stops the chef drifting into weird food nobody asked for. That last part is the reinforcement-learning step. DPO is a shortcut: skip building the scorecard and the slow trial loop, and learn straight from the 'this dish beats that dish' pairs in one pass.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The classical RLHF pipeline is one of the most-asked alignment questions because it is easy to recite the three stages and easy to get the ordering and the dependencies wrong. The recipe was popularised by InstructGPT in 2022 and remains the mental model behind almost every aligned chat model shipped since. When an interviewer asks where SFT and DPO sit, they are testing whether you understand the pipeline as a sequence of dependencies or as three memorised labels.
The trap the question sets is the relationship between SFT, the reward model, PPO, and DPO. Many candidates can list the stages but cannot say why they run in that order, what each stage consumes from the previous one, or where DPO inserts itself. A weak answer recites 'SFT, reward model, PPO' and stops. A strong answer explains that each stage is built from the artifact the previous stage produced, which is exactly why you cannot reorder them.
The other half of the question is DPO. Direct preference optimisation arrived as a simpler alternative to the reward-model-plus-PPO machinery, and a surprising number of candidates mis-state what it actually replaces. The clean answer is that DPO leaves stage 1 alone and folds stages 2 and 3 into a single training pass. It does not remove the supervised stage, and it does not remove the reference policy; it removes the explicit reward model and the online reinforcement-learning loop.
This deep dive walks each stage in turn, explains why the ordering is forced rather than conventional, then shows precisely how DPO collapses two stages into one while leaving the supervised stage untouched. The single sentence to anchor everything: SFT comes first in both pipelines, and the SFT checkpoint is the reference that everything downstream leans on.
Stage 1: supervised fine-tuning
Supervised fine-tuning starts from the pretrained base model, which can predict plausible text but does not reliably follow instructions. You assemble a dataset of prompt-and-demonstration pairs where humans write or curate the ideal response across a wide range of instructions. Training is ordinary next-token prediction on those demonstrations, the same objective used in pretraining but on a much narrower, higher-quality distribution.
The output is an instruction-following model. It is not yet preference-aligned, but it answers in roughly the right format, stays on topic, and refuses obviously bad requests. The dataset is small relative to pretraining, often tens of thousands of examples, because the goal is to shift behaviour toward following instructions rather than to teach new facts. Quality and diversity of the demonstrations matter far more than raw volume.
Why this stage must come first is the heart of the question. The reward model is initialised from this checkpoint and trained on its outputs. PPO uses the SFT model as both the starting policy and the KL anchor. DPO uses it as the reference policy in its log-ratio loss. Skip SFT and you have nothing competent to sample from, score against, or regularise toward. A raw base model produces such poor on-distribution samples that preference collection and optimisation both become unstable. That is why SFT is a prerequisite, not an optional warm-up, and why it is stage 1 regardless of whether you finish with PPO or DPO.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Stage | Classical RLHF | DPO pipeline |
|---|---|---|
| Stage 1 | SFT on demonstrations | SFT on demonstrations (same) |
| Stage 2 | Train a reward model on preference pairs | Folded into the DPO loss |
| Stage 3 | PPO against reward model with KL penalty | Folded into the DPO loss |
| Explicit reward model | Yes, a separate scalar-head model | No, the reward is implicit |
| Online sampling loop | Yes, PPO samples on-policy each step | No, offline on fixed preference pairs |
| Reference policy | SFT checkpoint anchors the KL penalty | SFT checkpoint is the reference in the log-ratio |
Real products, models, and research that use this idea.
- InstructGPT in 2022 established the canonical three-stage recipe: SFT, reward model, then PPO with a KL penalty against the SFT reference.
- Llama 4 post-training in 2026 runs SFT followed by preference optimisation, with DPO as the cheap default before any heavier online RL pass.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the reward model usually initialised from the SFT checkpoint rather than the raw base model?
Think about feature overlap and on-policy scoring. The reward model scores SFT-style outputs, so starting from SFT representations makes preference prediction easier and better calibrated to the policy's distribution.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying DPO replaces SFT. It does not. SFT is stage 1 in both pipelines, and the SFT checkpoint is the reference policy DPO trains against.
60 second bullets to scan on the way to the call.
The three classical stages in order after pretraining
Why SFT must come before reward modelling and PPO
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.