Name the three fields every pairwise preference record needs

**Three fields: prompt, chosen response, rejected response.** The prompt is the input both responses were generated for. Quality is always conditional on the prompt; a response that is great for one prompt may be off-topic for another. Without the prompt, the comparison is meaningless. The chosen response is the one the annotator (human or AI labeler) preferred. The rejected response is the other one. The labels are relative, not absolute: the record does not say how much better chosen was, only that it was preferred. The reward model trains on thousands of such records with a Bradley-Terry style loss. The model learns to assign a scalar score to any (prompt, response) pair such that for any training record the chosen response scores higher than the rejected one in expectation. This simple structure powers all standard RLHF, DPO, and related preference-optimization pipelines. More elaborate variants (KTO with thumbs-up/thumbs-down, RLAIF with constitutional principles) modify the labeling process but the core record shape stays the same.

**The three fields:** prompt, chosen response, rejected response. **The prompt anchors the comparison.** Quality is conditional on the prompt. A long technical answer is great for a debugging question and terrible for a greeting. The prompt is what makes the chosen vs rejected verdict interpretable. **Chosen and rejected are relative labels.** The record encodes that the labeler preferred chosen over rejected on this prompt. There is no absolute quality score, no preference strength, no fine-grained ranking. Just one bit of relative information per pair. **The Bradley-Terry loss.** The reward model `r(prompt, response)` is trained so that `sigma(r(prompt, chosen) - r(prompt, rejected))` is maximized in expectation. Mathematically: $$\mathcal{L} = -\mathbb{E}[\log \sigma(r(x, y_c) - r(x, y_r))]$$ where `x` is the prompt, `y_c` is chosen, `y_r` is rejected. The model learns a scalar reward function from thousands of such pairs without ever seeing absolute scores. **Variants that change the record shape.** KTO uses single thumbs-up or thumbs-down labels rather than pairs. ORPO works with both formats. DPO uses the exact same three-field record as classical RM-plus-PPO RLHF, just trained differently. Constitutional RLAIF replaces the human labeler with an AI labeler scoring against constitutional principles, but the three fields are unchanged.