**A reward model** in RLHF is a separate neural network whose job is to score how good a response is. It does this with one scalar number, given the prompt and the response together as input.
Architecturally it usually starts from the same base as the policy model, with the final language-modelling head replaced by a single linear layer that produces one number. So instead of predicting a probability distribution over tokens, it predicts one real-valued score per response.
The training data is preference pairs: a prompt plus two candidate responses, with a human label marking which response was preferred. The loss is the **Bradley-Terry loss**: the model is trained so that the score it assigns to the chosen response is higher than the score it assigns to the rejected one. The exact gap does not matter; only the ordering does.
This is why the reward model only needs pairwise comparison data, not gold answers or absolute scores. Humans are much more reliable at picking which of two responses they prefer than at scoring a single response on a quality scale, and the Bradley-Terry loss exploits exactly that property.
**A reward model** is the bridge in RLHF between human preferences and the policy gradient. It is a sequence-level scorer that takes a `(prompt, response)` pair and outputs one scalar reward.
**Architecture.** Typically initialised from the same SFT checkpoint that will become the policy, with the language-modelling head swapped for a single-output linear head. This gives the reward model strong language understanding and matches the policy's token distribution out of the box.
**Training data.** Human preference pairs: a shared prompt plus two candidate responses, with a label marking which was preferred. The reward model never sees an absolute quality score; it only sees pairwise rankings.
**Loss.** The Bradley-Terry loss, derived from the Bradley-Terry preference model:
$$\mathcal{L}_{\text{RM}} = -\log \sigma\big(r(x, y_w) - r(x, y_l)\big)$$
where `y_w` is the chosen response and `y_l` is the rejected one. The model is trained so the chosen scores higher than the rejected by any margin; the absolute scale is free.
**Where it plugs in.** Once trained, the reward model is frozen and used as the reward function during PPO. The policy generates responses, the reward model scores them, and a clipped policy-gradient update pushes the policy toward higher-reward outputs, with a KL penalty to the SFT reference to prevent drift.
**Why this design.** Humans find pairwise comparisons far more reliable than absolute ratings, so collecting preference data scales better than collecting scored data. The Bradley-Terry formulation respects that, requiring only ordering information from the labellers.
Quick reference
TL;DR
A reward model maps (prompt, response) to a scalar score and is trained on preference pairs under a Bradley-Terry loss so chosen beats rejected.
Imagine a restaurant critic who is shown two dishes from the same kitchen for the same order. The critic never says either dish is perfect or wrong; they just point at the one they preferred. Do this a few thousand times and a junior trainee can learn to predict which dish the critic would pick before the critic even tastes it. That trainee is the reward model. It does not learn an absolute standard of goodness; it learns the critic's relative preference from many head to head comparisons. Once trained, the trainee can score any new dish on a quiet scale that reliably ranks better above worse, which is exactly what the next stage of training needs.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You have a programmatically verifiable reward like unit tests passing or a math grader Real verifiable rewards beat learned reward models; the proxy is unnecessary when the actual signal is available.
You want to skip the PPO loop entirely and run direct preference optimisation DPO derives a closed-form loss that eliminates the reward model; if DPO covers your needs, training a separate RM is wasted effort.
Your preference dataset is tiny (under a few thousand pairs) and noisy Reward models need enough preference data to generalise; small noisy datasets produce poorly calibrated rewards that hurt PPO more than they help.
Quick scan · 2–4 min
Everything important, quickly.
6 min: the reward model's role + architecture + Bradley-Terry loss + why pairwise beats absolute + where it plugs into PPO + reward hacking + how DPO eliminates it.
Side-by-side
Reward signal
Input shape
Training signal
RLHF reward model
(prompt, response)
Pairwise preferences, Bradley-Terry
BLEU / ROUGE
(response, gold reference)
Surface overlap with reference
Verifiable reward
(response, checker)
Programmatic pass/fail
LLM judge
(prompt, response)
Sampled from a strong model
Real-world examples
Real products, models, and research that use this idea.
1OpenAI's original InstructGPT pipeline trained a reward model from the SFT checkpoint and ran PPO against it, the canonical RLHF setup.
2Anthropic's HH-RLHF reward model is publicly described in the Helpful and Harmless paper and uses Bradley-Terry on preference pairs.
3Llama 4 Maverick post-training combines a reward model with rule-based and verifiable rewards for parts of its alignment pipeline.
4DeepSeek V4's RL stage uses learned reward models alongside programmatic checkers for code and math.
5Hugging Face TRL provides RewardTrainer with Bradley-Terry loss as a standard component for community RLHF runs.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the Bradley-Terry loss only depend on the difference in scores?
Sign in or go Pro to see the approach for this follow-up.
QWhat is reward hacking, and how does PPO try to prevent it?
Sign in or go Pro to see the approach for this follow-up.
QWhy might you retrain the reward model midway through PPO?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming the reward model needs gold answers or absolute scores. It learns purely from pairwise preferences; the absolute reward scale is free and meaningless on its own.