Where do SFT and DPO sit in the classical RLHF pipeline? · Zenaique

Attempt it

Describe the three stages of the classical RLHF pipeline (as in InstructGPT / Ouyang et al. 2022). Then explain exactly where SFT and DPO sit, and how DPO simplifies the picture.

Free · 2 AI evals / day

Quick reference

TL;DR

Classical RLHF is three stages: SFT, then a reward model on preferences, then PPO against it with a KL penalty. SFT comes first in both; DPO folds the last two into one preference loss.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine training a new chef. First you show the chef many good dishes and say 'cook like this': that is supervised fine-tuning, the chef learns to follow instructions. Then you set two of the chef's dishes side by side and a taster says which is better, over and over; from those judgements you build a scorecard that predicts what tasters like. Finally the chef keeps cooking, you score each dish with the card, and the chef adjusts to score higher, while a rule stops the chef drifting into weird food nobody asked for. That last part is the reinforcement-learning step. DPO is a shortcut: skip building the scorecard and the slow trial loop, and learn straight from the 'this dish beats that dish' pairs in one pass.

Key concepts

RLHF pipeline Supervised fine-tuning Reward model PPO with KL penalty Direct Preference Optimization

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: name the three classical stages in order, explain why SFT must come first, describe reward model data and the PPO KL penalty, then show how DPO collapses stages 2 and 3 while keeping SFT as the reference.

Side-by-side

Stage	Classical RLHF	DPO pipeline
Stage 1	SFT on demonstrations	SFT on demonstrations (same)
Stage 2	Train a reward model on preference pairs	Folded into the DPO loss
Stage 3	PPO against reward model with KL penalty	Folded into the DPO loss
Explicit reward model	Yes, a separate scalar-head model	No, the reward is implicit
Online sampling loop	Yes, PPO samples on-policy each step	No, offline on fixed preference pairs
Reference policy	SFT checkpoint anchors the KL penalty	SFT checkpoint is the reference in the log-ratio

Real-world examples

Real products, models, and research that use this idea.

InstructGPT in 2022 established the canonical three-stage recipe: SFT, reward model, then PPO with a KL penalty against the SFT reference.
Llama 4 post-training in 2026 runs SFT followed by preference optimisation, with DPO as the cheap default before any heavier online RL pass.
DeepSeek V4 alignment combines an SFT stage with rule-based and preference-based RL, showing that SFT remains the shared prerequisite across recipes.
Hugging Face TRL ships SFTTrainer, RewardTrainer, PPOTrainer, and DPOTrainer, mirroring exactly where each stage sits in the pipeline.
Zephyr and many open-weight chat models popularised the SFT-then-DPO recipe, skipping a separate reward model and PPO entirely.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the reward model usually initialised from the SFT checkpoint rather than the raw base model?

Sign in or go Pro to see the approach for this follow-up.

QWhat exactly does the KL penalty in PPO regularise against, and what breaks without it?

Sign in or go Pro to see the approach for this follow-up.

QDerive intuitively why DPO needs no explicit reward model.

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Saying DPO replaces SFT. It does not. SFT is stage 1 in both pipelines, and the SFT checkpoint is the reference policy DPO trains against.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.