Why do frontier labs run multiple iterative RLHF rounds instead of one-shot alignment?
Why do frontier labs run multiple iterative RLHF rounds instead of one-shot alignment?
Labs run iterative RLHF rounds because each improved model reveals new edge cases that need fresh preference data and retuning.
Imagine teaching a student with weekly mock interviews. After week one, they fix easy mistakes, but now harder mistakes show up. If you stop coaching after one week, they plateau. RLHF is similar: one training round improves behavior, then exposes new weaknesses. Teams collect new comparisons on those harder cases and run another round. The cycle keeps moving the frontier instead of freezing at first-pass quality. Old practice tests also stop helping, because the student already passes them. The coach needs new tests aimed at the mistakes the student makes now.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
One round of RLHF trains a reward model on human comparisons of the current model's outputs, then optimizes a policy against that reward model. The round changes the thing it measured. The new policy writes differently, and the reward model is now scoring text unlike the text it learned from.
Iterative RLHF closes that gap by repeating the cycle: sample from the latest policy, collect fresh comparisons on those samples, retrain the reward model, and train the next policy. Public reports from Meta and Anthropic describe this loop in detail, and they show both why it works and what it costs.
Why one round goes stale
A reward model is a classifier trained on a distribution: pairs of responses that some earlier model produced. Its accuracy is highest on that distribution and lower elsewhere. RL then moves the policy toward responses the reward model scores highly, which is also a move away from the training distribution. After enough optimization, the policy samples mostly from regions where the reward model has seen little data.
The Llama 2 report states the problem directly. Improving Llama 2-Chat shifted the model's data distribution, and reward model accuracy can quickly degrade if it is not exposed to the new sample distribution. Collecting new preferences with the latest model, the report says, helps keep the reward model on-distribution and its reward accurate for the latest model.
Anthropic's HH-RLHF work saw the same effect from the other side. Its preference models became less calibrated at higher scores, and RLHF became less robust as the policy reached those scores. These are the scores a well-trained policy produces most often. After a single round, the policy keeps optimizing against a reward model that is weakest exactly where the policy now samples. That is the setup for reward hacking.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 2 trained RLHF-V1 through V5 on weekly batches of human preference data (1,418,091 Meta comparisons in total), collecting new preferences from the latest model each time. It used rejection sampling alone until V4, then PPO on top.
- Anthropic's HH-RLHF paper ran an iterated online mode in which preference models and RL policies were updated weekly with fresh human feedback. This improved the models as judged by crowdworkers.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect that the reward model has gone stale partway through a round?
Hold out fresh human comparisons on current policy samples and compare reward model accuracy on them with its original validation accuracy; watch for reward rising while human preference flattens.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
One-shot RLHF usually overfits first-wave mistakes and misses the harder failure modes that appear only after initial alignment gains.
60 second bullets to scan on the way to the call.
Why alignment data is non-stationary
Data flywheel mechanics
Primary sources. Browse if you want the original framing.
- Touvron et al., Llama 2: Open Foundation and Fine-Tuned Chat Models
- Bai et al., Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Llama Team, The Llama 3 Herd of Models
- Yuan et al., Self-Rewarding Language Models
- Ouyang et al., Training language models to follow instructions with human feedback
Same topic, related formats. Practice these next.