How do you collect preference pairs that improve alignment?
How do you collect preference pairs that improve alignment?
Collect preference pairs from diverse prompts, two on-policy responses that differ, written guidelines with a tie option, randomized order, measured annotator agreement, and fresh rounds.
Imagine teaching a new chef what diners like. You give a taster two plates of the same dish and ask which one is better. If every test is soup, the chef learns nothing about desserts. If both plates are identical, the taster's choice is a coin flip. If the taster always picks the plate on the left, you learn about the taster, not the food. So you test many dishes, make the two plates differ, swap left and right at random, write down what "better" means, and check that two tasters mostly agree. As the chef improves, you test the chef's newest plates, not last month's. Preference pairs work the same way: the prompt is the dish, the two answers are the plates, and the label is the taster's choice.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A preference pair is the basic unit of RLHF and DPO data: one prompt, two responses, and a human judgment of which response is better. A reward model trained on many such pairs becomes the scorer that RL optimizes, and DPO trains on the pairs directly. Either way, the model learns whatever separates the chosen response from the rejected one.
That fact drives every collection decision. If chosen responses are longer, the model learns length. If the prompts are all short questions, the model has no learned view of long chats. If labelers disagree on a third of the pairs, a third of the signal is noise. Good collection controls four things: which prompts you ask, which two responses you compare, how labelers judge them, and how you check and refresh the data.
What one preference record holds
A record has three required parts and several useful extras. The required parts are the prompt (with any earlier conversation turns), the two responses, and the choice. The extras are a preference strength, a tie or unsure option, a safety label, the labeler ID, the model version behind each response, and the order in which the responses were shown. The extras cost little at labeling time and are very hard to add later.
The reward model uses only the difference between the two responses. With the standard Bradley-Terry loss, it raises the score of the chosen response y_w relative to the rejected response y_l:
No absolute score appears, only the gap. So any feature that tends to differ between chosen and rejected responses becomes a reason to raise the score. Some of those features are what you want: correct facts, followed instructions, safe handling of harmful requests. Others are accidents of collection: extra length, a confident tone, agreement with the user, or a response that happened to be shown first.
Llama 2 used the extra fields directly. Annotators rated each choice as significantly better, better, slightly better, or negligibly better/unsure. The reward model loss then added a margin that grows with the rating, so clearly separated pairs push the scores further apart. A strength scale also lets you filter or down-weight near ties later.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- InstructGPT: about 40 screened contractors ranked 4 to 9 outputs per prompt over 33k reward model training prompts. Training labelers agreed with each other 72.6% of the time.
- Llama 2-Chat: 1,418,091 Meta binary comparisons for helpfulness and safety, collected in weekly batches with a four-level preference strength and separate helpfulness and safety reward models.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy can ranking K responses per prompt overfit a reward model, and how did InstructGPT handle it?
Count the K(K-1)/2 correlated pairs each prompt produces and think about what shuffling them in as separate examples does within one epoch.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Sampling pairs from an old model or a narrow prompt set and treating unchecked labels as ground truth, so the reward model learns collection bias instead of preference.
60 second bullets to scan on the way to the call.
Fields stored in one preference record
Sources and domain balance of the prompt set
Primary sources. Browse if you want the original framing.
- Ouyang et al., Training language models to follow instructions with human feedback
- Touvron et al., Llama 2: Open Foundation and Fine-Tuned Chat Models
- Bai et al., Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Singhal et al., A Long Way to Go: Investigating Length Correlations in RLHF
- Cui et al., UltraFeedback: Boosting Language Models with Scaled AI Feedback
Same topic, related formats. Practice these next.