Preference pair: name the two pieces and what the label expresses.

Attempt it

One preference example contains a shared prompt plus two candidate responses to that prompt: one tagged 'chosen' (preferred) and one tagged 'rejected' (less preferred). The label is relative, not absolute: it does not say either response is correct or wrong, only that a human rater (or a judge model) preferred the chosen one over the rejected one for that prompt. Downstream losses (Bradley-Terry reward modelling, DPO, KTO-paired, IPO) all consume this same {prompt, chosen, rejected} triple.

Your answer

Quick reference

TL;DR

A preference pair is one prompt plus two responses, one chosen and one rejected. The label is relative ordering, not absolute correctness.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine two restaurants serve you the same dish on the same evening. You are not asked which dish is objectively good; you are only asked which one you preferred tonight. That single comparison is the preference pair. There is no grade sheet, no answer key, no rubric saying either plate was right or wrong. The teaching signal is just the ordering between the two. Training systems that learn from these comparisons (the most famous is DPO) only need that ranking. They never claim the chosen plate was perfect or the rejected one was inedible. The labeller's job is simple: pick one of two, and the model figures out the rest from many such picks.

Key concepts

Preference learning RLHF DPO Reward model

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: the three fields of a preference pair + why the label is relative + how the data gets generated + which losses consume the triple + length bias and tie handling.

Real-world examples

Real products, models, and research that use this idea.

Anthropic's HH-RLHF dataset (Helpful and Harmless) is the canonical open preference-pair corpus, prompts plus chosen/rejected pairs labelled by humans.
UltraFeedback collects preference pairs at scale using GPT-4-level judge models, used to train many open-weight DPO models including Zephyr and Tulu.
Llama 4 Maverick and DeepSeek V4 post-training pipelines consume preference triples for DPO and PPO stages after SFT.
Hugging Face TRL's DPOTrainer expects exactly the {prompt, chosen, rejected} schema, with column names configurable.
OpenAssistant's open conversation dataset includes per-message rankings that are converted into preference pairs for community alignment runs.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow are the two responses in a preference pair usually generated?

Sign in or go Pro to see the approach for this follow-up.

QWhy does length bias show up in preference datasets, and how do you fight it?

Sign in or go Pro to see the approach for this follow-up.

QCan a preference pair encode a tie?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Reading the chosen response as the gold answer the model must memorise. The label only encodes ordering; the chosen response can still be flawed in absolute terms, just less flawed than the rejected one.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.