Why is human preference labeling the bottleneck in RLHF at scale?
Human preference labeling is the RLHF bottleneck because every pair needs expensive human judgment, disagreement handling, and ongoing relabeling as models improve.
Imagine training a chef, but every dish must be tasted by two experts before you can score it. You can cook thousands of dishes quickly, but expert tasting is slow and costly. RLHF works the same way: models can generate outputs cheaply, but preference labels come from people comparing answers. You also need quality checks when reviewers disagree, and you need fresh labels as the model gets better. That human review loop is why labeling speed, not GPU speed, usually caps RLHF throughput.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Every stage of RLHF except one runs on machines. Sampling responses, training the reward model and running PPO or DPO all scale with GPUs. The preference label is different: a trained person reads two or more responses and decides which is better. That step scales with human hours, and human hours do not grow overnight.
The cost is not only the price per click. Each label needs a screened rater, a rubric, checks for disagreement, and a fresh batch every round because the policy keeps changing. Those four costs together make labeling the slowest and most expensive part of the loop.
What one preference label costs
A preference label looks like one click: response A or response B. The click is the cheap part. Before it, the rater reads the prompt and both responses in full. In Llama 2's own preference data, an example averaged about 3.9 dialogue turns and about 800 tokens. A coding or medical prompt also needs a rater who can check whether the code runs or the advice is safe. Those raters are scarce and expensive.
The rater also has to be found and trained. InstructGPT hired about 40 contractors and selected them with a screening test on how well they spotted harmful outputs and matched the researchers' judgments. Every rubric change means retraining the pool. Richer formats raise the per-prompt effort further. InstructGPT raters ranked 4 to 9 responses per prompt. Llama 2 raters also chose one of four preference strengths, from "significantly better" to "negligibly better or unsure".
So the unit of cost is human reading time per comparison, multiplied by the expertise the prompt demands. Generation cost per response falls as inference gets cheaper; reading cost per response does not.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- InstructGPT hired about 40 contractors after a screening test, had them rank 4 to 9 responses per prompt, and reports 72.6% agreement among its training labelers.
- Llama 2 collected over 1 million binary comparisons in weekly batches, with a four-level preference strength scale. It refreshed its reward models across RLHF versions V1 to V5.
What an interviewer would ask next. Try answering before peeking at the approach.
QLabelers agree with each other only 73% of the time. What does that mean for the best accuracy a reward model can show on held-out human labels?
If a second human matches the first only 73% of the time, a reward model scored against one human label per pair also hits a noise ceiling. Think about overlap labeling and removing ties or low-confidence pairs.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Teams often budget for GPU time but underestimate reviewer disagreement, quality audits, and relabeling rounds that dominate RLHF timelines.
60 second bullets to scan on the way to the call.
What one preference label costs in human time
Labeler agreement figures and the noise ceiling they imply
Primary sources. Browse if you want the original framing.
- Ouyang et al., Training language models to follow instructions with human feedback
- Touvron et al., Llama 2: Open Foundation and Fine-Tuned Chat Models
- Bai et al., Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Lee et al., RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
- Bai et al., Constitutional AI: Harmlessness from AI Feedback
Same topic, related formats. Practice these next.