- 1Train a reward model on pairwise preference judgments
- 2Run supervised fine-tuning on high quality demonstrations
- 3Optimize the policy with PPO against reward while anchoring with KL
Canonical RLHF order is SFT first, then reward-model training, then PPO policy optimization with KL anchoring to preserve baseline behavior.
Think of coaching a player in three rounds. First you teach the basics. Second you build a judge who can score good versus bad moves. Third you let the player practice while using that judge and a safety rope so they do not drift too far from fundamentals. RLHF follows the same order: SFT basics, reward-model judge, then RL optimization with a KL rope to avoid unstable behavior.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Most interview answers about the canonical RLHF stage order from base model to aligned assistant are technically correct but operationally shallow. They name one formula, then stop before discussing how data quality, metric choice, and optimization pressure determine whether the system actually improves user outcomes. In real post-training pipelines, that missing middle is where most failures happen.
The core question here is why the sequence SFT -> reward model -> policy optimization is structurally necessary. To answer it well, you need to connect mechanism to deployment reality: what signal is learned, why that signal can drift, and which guardrails keep optimization honest. This deep dive walks from foundations to production checks so the concept is not just memorized, but usable in design reviews and interview discussions.
Mechanism and objective: what is actually optimized
Start with the optimization target, because confusion here causes downstream mistakes. In this topic, the learning loop is built around capability prior from pretraining, instruction grounding from SFT, preference signal from RM, and constrained optimization in RL. That list sounds simple, but each element constrains what the model can and cannot learn. If you are clear on the target signal, many design choices become obvious instead of hand-wavy.
A useful interview move is to separate absolute quality from relative preference. Many alignment objectives do not teach a universal quality score; they teach ordering under specific label policies. That means calibration, coverage, and disagreement handling are first-class concerns, not afterthoughts. When teams forget this, they celebrate metric gains that fail to transfer to users.
The mathematical form below captures the mechanism compactly. Treat it as a map of assumptions: if labels are noisy, if distributions shift, or if optimization pressure is too strong, the same equation can still produce poor behavior. The formula is necessary for precision, but governance around it is what keeps the system useful.
Pretrain \rightarrow SFT \rightarrow RM \rightarrow RLSituations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- InstructGPT described this exact sequence: supervised fine-tuning, reward modeling, then PPO fine-tuning.
- Most open RLHF training stacks preserve the same stage order even when swapping PPO for DPO-like objectives.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does DPO change this three-stage order?
Explain that DPO can replace PPO and explicit reward-model stage, but still relies on strong supervised baseline and preference data.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reordering stages breaks data dependencies and makes optimization unstable or inefficient.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.