What is RLHF, and why is it used after pretraining?

Attempt it

Quick reference

TL;DR

RLHF, Reinforcement Learning from Human Feedback, trains a reward model on human preference labels and uses RL to align the LLM with what humans prefer.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine training a comedian. Pretraining is them reading every joke book ever written, they know lots of jokes but they don't know which ones land. Then you put them on stage in front of an audience and have the audience pick between pairs of jokes: this one or that one, funnier. You train a small judge that learned the audience's taste from all those picks. Now you put the comedian back on stage and they tell jokes, the judge scores each one, and the comedian adjusts to score higher. That's RLHF. The audience is the human labelers. The judge is the reward model. The comedian getting tuned is the LLM. Pretraining gives the model knowledge; RLHF teaches it which responses humans actually want.

Key concepts

Reward model Fine-tuning Proximal Policy Optimization

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: three pipeline stages + why pretraining alone is insufficient + role of reward model and KL leash + DPO and RLAIF as 2026 variants + behavior vs knowledge distinction.

Real-world examples

Real products, models, and research that use this idea.

InstructGPT (2022) was the OpenAI paper that demonstrated PPO-based RLHF and powered the first ChatGPT.
Anthropic's constitutional AI approach uses RLAIF (AI feedback against a constitution) instead of human pairs for much of the alignment signal.
Llama 3 and Llama 4 instruct models use a multi-stage pipeline combining DPO, rejection sampling, and RLAIF rather than vanilla PPO.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat goes wrong without the KL leash to the SFT reference?

Sign in or go Pro to see the approach for this follow-up.

QHow does RLAIF differ from RLHF and when would you use it?

Sign in or go Pro to see the approach for this follow-up.

QHow would you eval whether RLHF is actually improving the model?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Saying RLHF teaches the model new facts. It teaches behavior preference, helpfulness, harmlessness, format, not new world knowledge.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.