The prompt, the chosen (preferred) response, and the rejected response. The reward model is trained with a Bradley-Terry style loss to score the chosen response above the rejected one for that prompt; thousands of such pairs teach it a general quality ranking.
**Three fields: prompt, chosen response, rejected response.**
The prompt is the input both responses were generated for. Quality is always conditional on the prompt; a response that is great for one prompt may be off-topic for another. Without the prompt, the comparison is meaningless.
The chosen response is the one the annotator (human or AI labeler) preferred. The rejected response is the other one. The labels are relative, not absolute: the record does not say how much better chosen was, only that it was preferred.
The reward model trains on thousands of such records with a Bradley-Terry style loss. The model learns to assign a scalar score to any (prompt, response) pair such that for any training record the chosen response scores higher than the rejected one in expectation.
This simple structure powers all standard RLHF, DPO, and related preference-optimization pipelines. More elaborate variants (KTO with thumbs-up/thumbs-down, RLAIF with constitutional principles) modify the labeling process but the core record shape stays the same.
**The three fields:** prompt, chosen response, rejected response.
**The prompt anchors the comparison.** Quality is conditional on the prompt. A long technical answer is great for a debugging question and terrible for a greeting. The prompt is what makes the chosen vs rejected verdict interpretable.
**Chosen and rejected are relative labels.** The record encodes that the labeler preferred chosen over rejected on this prompt. There is no absolute quality score, no preference strength, no fine-grained ranking. Just one bit of relative information per pair.
**The Bradley-Terry loss.** The reward model `r(prompt, response)` is trained so that `sigma(r(prompt, chosen) - r(prompt, rejected))` is maximized in expectation. Mathematically:
$$\mathcal{L} = -\mathbb{E}[\log \sigma(r(x, y_c) - r(x, y_r))]$$
where `x` is the prompt, `y_c` is chosen, `y_r` is rejected. The model learns a scalar reward function from thousands of such pairs without ever seeing absolute scores.
**Variants that change the record shape.** KTO uses single thumbs-up or thumbs-down labels rather than pairs. ORPO works with both formats. DPO uses the exact same three-field record as classical RM-plus-PPO RLHF, just trained differently. Constitutional RLAIF replaces the human labeler with an AI labeler scoring against constitutional principles, but the three fields are unchanged.
Quick reference
TL;DR
Prompt, chosen response, rejected response. The reward model uses Bradley-Terry loss to score chosen above rejected on the same prompt.
Think of a pizza tasting where you write down: what was ordered (the prompt), the pizza everyone liked best (chosen), and the pizza that came in second (rejected). After thousands of these notes, a judge can learn what people prefer in general, even for pizzas they have not tasted. The reward model is that judge. It does not need to know how much better the winner was, only that on this prompt the chosen response was preferred over the rejected one.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
KTO-style training where single-response thumbs labels are the input The record shape is different (prompt, response, thumbs) and the loss does not need pairs
RLVR with verifiable rewards (math, code unit tests) The reward is programmatic; no preference pairs are collected at all
Pure SFT with no preference signal SFT consumes (prompt, ideal response) pairs, not preference comparisons
Quick scan · 2–4 min
Everything important, quickly.
5 minutes: the three fields, the Bradley-Terry loss, why minimal is good, and the sibling formats (DPO, KTO, ORPO).
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's HH-RLHF dataset stores exactly these three fields per record and is one of the canonical open preference corpora
2OpenAI's WebGPT and InstructGPT preference data follow the same three-field structure
3Meta's Llama 3 and Llama 4 post-training pipelines consume preference records in the same shape, used for iterative DPO rounds
4Stanford's UltraFeedback dataset uses the three-field structure with AI-generated chosen/rejected pairs scored by GPT-4 class judges
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does adding a preference-strength field often degrade the RM rather than improve it?
Sign in or go Pro to see the approach for this follow-up.
QHow does DPO reuse the same three-field record without a separate reward model?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Forgetting that the prompt anchors the comparison. A chosen/rejected pair without its prompt is meaningless because quality is conditional on the prompt.