Reward model in RLHF: what does it score, and on what data does it train?

MCQ·Easy·4.0 · 0·~1 min

Attempt it

**A reward model** in RLHF is a separate neural network whose job is to score how good a response is. It does this with one scalar number, given the prompt and the response together as input. Architecturally it usually starts from the same base as the policy model, with the final language-modelling head replaced by a single linear layer that produces one number. So instead of predicting a probability distribution over tokens, it predicts one real-valued score per response. The training data is preference pairs: a prompt plus two candidate responses, with a human label marking which response was preferred. The loss is the **Bradley-Terry loss**: the model is trained so that the score it assigns to the chosen response is higher than the score it assigns to the rejected one. The exact gap does not matter; only the ordering does. This is why the reward model only needs pairwise comparison data, not gold answers or absolute scores. Humans are much more reliable at picking which of two responses they prefer than at scoring a single response on a quality scale, and the Bradley-Terry loss exploits exactly that property.

**A reward model** is the bridge in RLHF between human preferences and the policy gradient. It is a sequence-level scorer that takes a `(prompt, response)` pair and outputs one scalar reward. **Architecture.** Typically initialised from the same SFT checkpoint that will become the policy, with the language-modelling head swapped for a single-output linear head. This gives the reward model strong language understanding and matches the policy's token distribution out of the box. **Training data.** Human preference pairs: a shared prompt plus two candidate responses, with a label marking which was preferred. The reward model never sees an absolute quality score; it only sees pairwise rankings. **Loss.** The Bradley-Terry loss, derived from the Bradley-Terry preference model: $$\mathcal{L}_{\text{RM}} = -\log \sigma\big(r(x, y_w) - r(x, y_l)\big)$$ where `y_w` is the chosen response and `y_l` is the rejected one. The model is trained so the chosen scores higher than the rejected by any margin; the absolute scale is free. **Where it plugs in.** Once trained, the reward model is frozen and used as the reward function during PPO. The policy generates responses, the reward model scores them, and a clipped policy-gradient update pushes the policy toward higher-reward outputs, with a KL penalty to the SFT reference to prevent drift. **Why this design.** Humans find pairwise comparisons far more reliable than absolute ratings, so collecting preference data scales better than collecting scored data. The Bradley-Terry formulation respects that, requiring only ordering information from the labellers.

Quick reference

TL;DR

A reward model maps (prompt, response) to a scalar score and is trained on preference pairs under a Bradley-Terry loss so chosen beats rejected.

Memory aid

Easy to grasp

Imagine a restaurant critic who is shown two dishes from the same kitchen for the same order. The critic never says either dish is perfect or wrong; they just point at the one they preferred. Do this a few thousand times and a junior trainee can learn to predict which dish the critic would pick before the critic even tastes it. That trainee is the reward model. It does not learn an absolute standard of goodness; it learns the critic's relative preference from many head to head comparisons. Once trained, the trainee can score any new dish on a quiet scale that reliably ranks better above worse, which is exactly what the next stage of training needs.

Key concepts

Reward model RLHF Bradley-Terry model Preference learning

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

6 min: the reward model's role + architecture + Bradley-Terry loss + why pairwise beats absolute + where it plugs into PPO + reward hacking + how DPO eliminates it.

Side-by-side

Reward signal	Input shape	Training signal
RLHF reward model	(prompt, response)	Pairwise preferences, Bradley-Terry
BLEU / ROUGE	(response, gold reference)	Surface overlap with reference
Verifiable reward	(response, checker)	Programmatic pass/fail
LLM judge	(prompt, response)	Sampled from a strong model

Real-world examples

Real products, models, and research that use this idea.

OpenAI's original InstructGPT pipeline trained a reward model from the SFT checkpoint and ran PPO against it, the canonical RLHF setup.
Anthropic's HH-RLHF reward model is publicly described in the Helpful and Harmless paper and uses Bradley-Terry on preference pairs.
Llama 4 Maverick post-training combines a reward model with rule-based and verifiable rewards for parts of its alignment pipeline.
DeepSeek V4's RL stage uses learned reward models alongside programmatic checkers for code and math.
Hugging Face TRL provides RewardTrainer with Bradley-Terry loss as a standard component for community RLHF runs.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the Bradley-Terry loss only depend on the difference in scores?

QWhat is reward hacking, and how does PPO try to prevent it?

QWhy might you retrain the reward model midway through PPO?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming the reward model needs gold answers or absolute scores. It learns purely from pairwise preferences; the absolute reward scale is free and meaningless on its own.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.