Reward model in RLHF: what does it score, and on what data does it train?

MCQ·Easy·4.0 · 0·~1 min·Asked atAi21 Pinterest Replicate·Relevant atAnthropic Fireworks Ai Together Ai

Attempt it

**A reward model** in RLHF is a separate neural network whose job is to score how good a response is. It does this with one scalar number, given the prompt and the response together as input. Architecturally it usually starts from the same base as the policy model, with the final language-modelling head replaced by a single linear layer that produces one number. So instead of predicting a probability distribution over tokens, it predicts one real-valued score per response. The training data is preference pairs: a prompt plus two candidate responses, with a human label marking which response was preferred. The loss is the **Bradley-Terry loss**: the model is trained so that the score it assigns to the chosen response is higher than the score it assigns to the rejected one. The exact gap does not matter; only the ordering does. This is why the reward model only needs pairwise comparison data, not gold answers or absolute scores. Humans are much more reliable at picking which of two responses they prefer than at scoring a single response on a quality scale, and the Bradley-Terry loss exploits exactly that property.

**A reward model** is the bridge in RLHF between human preferences and the policy gradient. It is a sequence-level scorer that takes a `(prompt, response)` pair and outputs one scalar reward. **Architecture.** Typically initialised from the same SFT checkpoint that will become the policy, with the language-modelling head swapped for a single-output linear head. This gives the reward model strong language understanding and matches the policy's token distribution out of the box. **Training data.** Human preference pairs: a shared prompt plus two candidate responses, with a label marking which was preferred. The reward model never sees an absolute quality score; it only sees pairwise rankings. **Loss.** The Bradley-Terry loss, derived from the Bradley-Terry preference model: $$\mathcal{L}_{\text{RM}} = -\log \sigma\big(r(x, y_w) - r(x, y_l)\big)$$ where `y_w` is the chosen response and `y_l` is the rejected one. The model is trained so the chosen scores higher than the rejected by any margin; the absolute scale is free. **Where it plugs in.** Once trained, the reward model is frozen and used as the reward function during PPO. The policy generates responses, the reward model scores them, and a clipped policy-gradient update pushes the policy toward higher-reward outputs, with a KL penalty to the SFT reference to prevent drift. **Why this design.** Humans find pairwise comparisons far more reliable than absolute ratings, so collecting preference data scales better than collecting scored data. The Bradley-Terry formulation respects that, requiring only ordering information from the labellers.

TL;DR

A reward model maps (prompt, response) to a scalar score and is trained on preference pairs under a Bradley-Terry loss so chosen beats rejected.

Memory aid

Easy to grasp

Imagine a restaurant critic who is shown two dishes from the same kitchen for the same order. The critic never says either dish is perfect or wrong; they just point at the one they preferred. Do this a few thousand times and a junior trainee can learn to predict which dish the critic would pick before the critic even tastes it. That trainee is the reward model. It does not learn an absolute standard of goodness; it learns the critic's relative preference from many head to head comparisons. Once trained, the trainee can score any new dish on a quiet scale that reliably ranks better above worse, which is exactly what the next stage of training needs.

Key concepts

Reward model RLHF Bradley-Terry model Preference learning

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The reward model is the load-bearing component of classical RLHF. It is the bridge between human preferences and the policy-gradient optimisation, and it embodies a specific bet: that human judgement can be captured as a scalar function of (prompt, response), learnable from pairwise comparison data. Understanding what the reward model is, how it is trained, and how it plugs into the larger RLHF pipeline is the foundation for understanding why DPO was invented to eliminate it.

This deep dive covers four things. First, what the reward model literally outputs and what its architecture looks like. Second, where the training data comes from and how the Bradley-Terry loss exploits its pairwise nature. Third, how the trained reward model is used as the reward signal during PPO. Fourth, the failure modes (reward hacking, distribution shift, length bias) that motivated the DPO-family alternatives.

The headline is simple. The reward model is a scalar regressor over (prompt, response), initialised from an SFT checkpoint with the LM head swapped for a one-output linear head. It is trained on human preference pairs under the loss -log sigma(r(chosen) - r(rejected)). Once trained, it is frozen and called inside the PPO loop to score the policy's generated responses. The whole machinery works because pairwise comparison is the most reliable feedback humans give, and the Bradley-Terry formulation lets ordering data identify an implicit absolute reward up to an additive constant.

Get the reward model right and the rest of RLHF is mechanical engineering. Get it wrong (badly calibrated, trained on biased pairs, or starved of in-distribution updates) and the PPO loop will faithfully optimise toward the wrong target, producing a policy that scores high on the broken reward model but underperforms with real humans.

What the reward model is and what it outputs

Architecture

The reward model is a neural network that takes a (prompt, response) pair as input and outputs a single scalar. The standard recipe initialises it from the same SFT checkpoint that will become the policy, then replaces the final language-modelling head (which produces a distribution over vocabulary) with a single-output linear layer that produces one number.

This design choice has two benefits. It inherits the language understanding of the SFT model out of the box, which means the reward model already knows what coherent English looks like. And it matches the policy's tokenisation and embedding space, so the reward predictions are well-calibrated against the policy's natural distribution.

Output semantics

The output is a real-valued scalar with no fixed range. It is not a probability. It is not bounded between 0 and 1. The absolute value of any single reward is meaningless on its own; only differences between rewards on the same prompt matter.

This is a direct consequence of the Bradley-Terry training: the loss only constrains the order, so the reward is identifiable only up to an additive constant. Treating the absolute scale as meaningful is one of the most common subtle errors in reward-model interpretation.

Compute economics

Reward models are full forward passes through a transformer, so they are expensive to call. Some teams use a smaller reward model than the policy to reduce per-step PPO cost. Others share weights between the policy and reward model (the value head architecture in some PPO implementations) at the cost of some quality.

Training data and the Bradley-Terry loss

How the reward model plugs into PPO

Failure modes and the DPO alternative

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Reward signal	Input shape	Training signal
RLHF reward model	(prompt, response)	Pairwise preferences, Bradley-Terry
BLEU / ROUGE	(response, gold reference)	Surface overlap with reference
Verifiable reward	(response, checker)	Programmatic pass/fail
LLM judge	(prompt, response)	Sampled from a strong model

Real products, models, and research that use this idea.

OpenAI's original InstructGPT pipeline trained a reward model from the SFT checkpoint and ran PPO against it, the canonical RLHF setup.
Anthropic's HH-RLHF reward model is publicly described in the Helpful and Harmless paper and uses Bradley-Terry on preference pairs.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the Bradley-Terry loss only depend on the difference in scores?

Start from the assumption that P(y_w preferred over y_l) = sigmoid(r(y_w) - r(y_l)). The cross-entropy of the observed pair under this assumption is -log sigmoid(r(y_w) - r(y_l)). The score is only identifiable up to an additive constant, which is why absolute reward scale is free.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Assuming the reward model needs gold answers or absolute scores. It learns purely from pairwise preferences; the absolute reward scale is free and meaningless on its own.

60 second bullets to scan on the way to the call.

Sketch the reward-model architecture and the role of its scalar head.
The training data shape: preference pairs

Primary sources. Browse if you want the original framing.