A preference pair is one prompt plus two responses, one chosen and one rejected. The label is relative ordering, not absolute correctness.
Imagine two restaurants serve you the same dish on the same evening. You are not asked which dish is objectively good; you are only asked which one you preferred tonight. That single comparison is the preference pair. There is no grade sheet, no answer key, no rubric saying either plate was right or wrong. The teaching signal is just the ordering between the two. Training systems that learn from these comparisons (the most famous is DPO) only need that ranking. They never claim the chosen plate was perfect or the rejected one was inedible. The labeller's job is simple: pick one of two, and the model figures out the rest from many such picks.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Preference pairs are the data primitive of modern alignment. Every method that nudges a model toward what humans like, DPO, PPO with a learned reward, KTO-paired, IPO, SimPO, ORPO, consumes the same {prompt, chosen, rejected} triple. Understanding what that triple represents, how it gets collected, and what its label actually says is the foundation for understanding any preference-learning method that builds on top.
The shape is deceptively simple. A prompt, two candidate responses, a binary label marking which response a rater preferred. There is no scale, no rubric, no per-dimension breakdown. Just an ordering. That minimalism is the source of the format's power: it scales cheaply, labels reliably, and slots into a wide family of training methods without modification.
This deep dive walks through what each field of a preference pair represents, why the label is relative rather than absolute, the three common recipes for generating the responses, the small library of losses that consume the triple, and the failure modes (length bias, position bias, distribution shift) that teams discover the hard way once they ship preference-based training.
The stakes matter. Misreading a preference pair as supervised data, training the model to literally copy the chosen response, is the most common interview blunder for this topic. Preference learning works precisely because it does not demand absolute correctness, it only demands ordering. Get this distinction right and the whole alignment stack opens up.
The three fields and what they encode
The shape
A preference pair has three fields:
prompt: the input the model would be asked to respond to. This is shared between the two candidates and is fed to the model (and the reward model) at training time.chosen: the response the rater preferred.rejected: the response the rater preferred less.
The two responses must be for the same prompt. The reward signal is conditional on the prompt: a response that is great for one question may be terrible for another, so the comparison only carries information when the prompt is held constant.
What the label actually says
The label is purely a binary ordering: chosen > rejected for this prompt, according to this rater. It does not claim:
- That
chosenis correct in any absolute sense. - That
rejectedis wrong. - That
chosenis good enough to ship. - That
chosenwould beat any third response not in the pair.
It only encodes the local ordering. The downstream loss aggregates many such orderings to recover a useful global signal, but the data itself stays minimalist.
Why this minimalism is the point
Pairwise comparison is the most reliable feedback humans give. Asked to score a response 1 to 5, raters disagree on what a 3 means and the same rater drifts over time. Asked which of two responses they prefer, raters are consistent and quick. The format is designed around that human-factor reality.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's HH-RLHF dataset (Helpful and Harmless) is the canonical open preference-pair corpus, prompts plus chosen/rejected pairs labelled by humans.
- UltraFeedback collects preference pairs at scale using GPT-4-level judge models, used to train many open-weight DPO models including Zephyr and Tulu.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow are the two responses in a preference pair usually generated?
Three patterns: sample the same SFT model twice at temperature and have a human pick, mix samples from a stronger and weaker model, or sample from the current policy in iterative DPO and judge with a reward model. Each pattern has different bias properties.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reading the chosen response as the gold answer the model must memorise. The label only encodes ordering; the chosen response can still be flawed in absolute terms, just less flawed than the rejected one.
60 second bullets to scan on the way to the call.
The three fields of a preference pair
Why both responses must share a prompt
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.