- 1Train a reward model on pairwise preference judgments
- 2Run supervised fine-tuning on high quality demonstrations
- 3Optimize the policy with PPO against reward while anchoring with KL
Canonical RLHF order is SFT first, then reward-model training, then PPO policy optimization with KL anchoring to preserve baseline behavior.
Think of coaching a player in three rounds. First you teach the basics. Second you build a judge who can score good versus bad moves. Third you let the player practice while using that judge and a safety rope so they do not drift too far from fundamentals. RLHF follows the same order: SFT basics, reward-model judge, then RL optimization with a KL rope to avoid unstable behavior. Skip the basics and the judge has only bad moves to score. Skip the judge and practice gets no feedback.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The standard RLHF pipeline has three stages: supervised fine-tuning (SFT) on demonstrations, reward model (RM) training on human comparisons, and PPO against the RM with a KL penalty. The order is fixed by data flow. Each stage consumes something that only the stage before it can produce.
InstructGPT, the paper that made this pipeline the standard for chat models, shows the dependencies plainly. Its method section starts the RM from the SFT model. Its comparisons come mostly from SFT outputs. Its PPO policy starts from the SFT model, its KL penalty is measured against the SFT model, and its value function starts from the RM.
Stage 1, SFT: from text predictor to assistant
A pretrained base model continues text. Give it a question and it may answer, or it may write three more questions, a forum thread or a list of links, because all of these are likely continuations on the web. It has the knowledge but not the habit of answering.
SFT adds the habit. Labelers write good responses to real prompts, and the model trains on these prompt and response pairs with the usual next-token loss on the response. InstructGPT used about 13k training prompts for this stage. The result is a policy that already answers in the right format and is roughly helpful.
SFT has to come first for three reasons:
- Comparisons need sensible candidates. Labelers in stage 2 compare outputs. Two rambling base model continuations give them little to judge, and their labels become noisy.
- RL needs a good starting point. PPO improves on what the policy already samples. If almost no sample is a good answer, the reward barely varies and the gradient carries little signal.
- The KL penalty needs a reference. The SFT model is the behavior that PPO is allowed to move away from only slowly.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- InstructGPT described this exact sequence: supervised fine-tuning on about 13k prompts, reward modeling on about 33k prompts, then PPO on about 31k prompts.
- Learning to summarize from human feedback (2020) used the same three steps for summarization: a supervised baseline, a reward model on human comparisons, then PPO with a KL penalty.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does DPO change this three-stage order?
Explain that DPO can replace PPO and the explicit reward-model stage, but still relies on a strong supervised baseline and preference data.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reordering stages breaks data dependencies and makes optimization unstable or inefficient.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.