Name the three fields every pairwise preference record needs
Prompt, chosen response, rejected response. The reward model uses Bradley-Terry loss to score chosen above rejected on the same prompt.
Think of a pizza tasting where you write down: what was ordered (the prompt), the pizza everyone liked best (chosen), and the pizza that came in second (rejected). After thousands of these notes, a judge can learn what people prefer in general, even for pizzas they have not tasted. The reward model is that judge. It does not need to know how much better the winner was, only that on this prompt the chosen response was preferred over the rejected one.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The pairwise preference record is the atomic unit of RLHF data. Three fields: prompt, chosen response, rejected response. That is enough to train a reward model, run DPO, do iterative preference optimization, or generate the labels for an RLAIF round. The minimality is a feature, not a limitation, and understanding why takes a small detour through Bradley-Terry preference modeling.
This deep dive walks through what each field encodes, why the prompt is mandatory, how the Bradley-Terry loss consumes the records, why adding fields usually hurts rather than helps, and how the modern variants (DPO, KTO, ORPO, constitutional RLAIF) relate to the same record shape.
The goal is to leave you able to describe preference data without ambiguity, recognize when a proposed schema deviates from the canonical shape and why, and place the variants on a clear map of who uses the same records and who uses different ones.
The three fields and what each encodes
Prompt. The input both responses were generated for. Quality is conditional on the prompt; the same response can be excellent for one prompt and terrible for another. The prompt is what makes the chosen vs rejected verdict interpretable as a quality judgment rather than as a context-free preference. A record without its prompt is unusable.
Chosen response. The response the labeler (human or AI) preferred. This is one of multiple candidates generated for the same prompt, typically by sampling the policy at varied temperatures or by comparing different policy checkpoints. The label encodes a relative preference, not an absolute score.
Rejected response. The other candidate. Again, the encoding is relative: the labeler picked chosen over rejected, but the record does not say how much better chosen was, whether the labeler considered chosen actually good or merely less bad, or what other candidates were considered.
The minimalism reflects what Bradley-Terry preference modeling actually consumes: one bit of relative information per pair. Anything more is operational metadata (rater ID, timestamps, position logs) used for auditing rather than for the loss.
The three fields anchor the entire preference-learning stack. Prompt is the context the responses are conditional on. Chosen is the response judged better. Rejected is the response judged worse. Everything else (annotator id, timestamp, confidence, rubric tags) is metadata. The minimum viable preference record is exactly this triple, and every major preference-learning method (DPO, PPO with a Bradley-Terry RM, IPO, KTO with a synthesis trick) consumes a dataset of triples.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's HH-RLHF dataset stores exactly these three fields per record and is one of the canonical open preference corpora
- OpenAI's WebGPT and InstructGPT preference data follow the same three-field structure
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does adding a preference-strength field often degrade the RM rather than improve it?
Strength judgments are noisier than the preference itself, and Bradley-Terry has no principled way to consume them. Including noise as if it were signal moves the loss surface in unhelpful directions.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forgetting that the prompt anchors the comparison. A chosen/rejected pair without its prompt is meaningless because quality is conditional on the prompt.
60 second bullets to scan on the way to the call.
The three core fields and why each is necessary
Why Bradley-Terry only uses relative preference, not strength
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.