What is RLHF, and why is it used after pretraining?
RLHF, Reinforcement Learning from Human Feedback, trains a reward model on human preference labels and uses RL to align the LLM with what humans prefer.
Imagine training a comedian. Pretraining is them reading every joke book ever written, they know lots of jokes but they don't know which ones land. Then you put them on stage in front of an audience and have the audience pick between pairs of jokes: this one or that one, funnier. You train a small judge that learned the audience's taste from all those picks. Now you put the comedian back on stage and they tell jokes, the judge scores each one, and the comedian adjusts to score higher. That's RLHF. The audience is the human labelers. The judge is the reward model. The comedian getting tuned is the LLM. Pretraining gives the model knowledge; RLHF teaches it which responses humans actually want.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
RLHF (Reinforcement Learning from Human Feedback) is the post training process that turned the raw next-token predictors of 2020-2021 into the helpful assistants of 2022 onward. The InstructGPT paper made it canonical; ChatGPT made it world changing.
The interview question 'what is RLHF and why is it used' has a clean three-stage answer (SFT, reward model, RL) and a richer underlying story (why pretraining alone is insufficient, what the KL leash does, how DPO and RLAIF reshape the picture).
This deep dive walks through the gap RLHF closes, the canonical pipeline, what each stage does, and the 2026 landscape where the term has broadened to cover several related techniques.
The gap RLHF closes
Pretraining optimizes one thing: predict the next token of internet text. The objective is cross-entropy on a giant token-prediction task, and it works astonishingly well at giving the model knowledge, world models, and a sense of how language flows.
It does not work at making the model helpful.
Next-token prediction rewards mimicry. If a pretrained model is asked 'what's the capital of France?', a high-likelihood completion might be 'Good question, let's see if you know...' because that quiz show pattern appeared in its training data. The model isn't broken, it's predicting what comes next in the text distribution. The text distribution doesn't reflect 'what a helpful assistant would say'.
RLHF closes this gap by introducing a different signal: human preference. Instead of optimizing 'what's likely to come next', the model optimizes 'what humans actually prefer to see'. The objective changes from imitation to alignment with stated preferences.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- InstructGPT (2022) was the OpenAI paper that demonstrated PPO-based RLHF and powered the first ChatGPT.
- Anthropic's constitutional AI approach uses RLAIF (AI feedback against a constitution) instead of human pairs for much of the alignment signal.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat goes wrong without the KL leash to the SFT reference?
The policy reward-hacks. Outputs that maximize the reward model's score drift far from natural language, repetitive, off-distribution, gibberish. KL keeps the policy in the SFT region where the reward model's predictions are calibrated.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying RLHF teaches the model new facts. It teaches behavior preference, helpfulness, harmlessness, format, not new world knowledge.
60 second bullets to scan on the way to the call.
The three pipeline stages and what each accomplishes
Why pretraining alone doesn't produce a helpful assistant
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.