Explain how a reward model is trained from pairwise human preferences

A reward model in RLHF (reinforcement learning from human feedback) is a scorer, not the chatbot users talk to. You collect pairs of outputs for the same prompt and ask annotators which one they prefer. The model then learns a scalar (single number) score for each output so preferred answers end up with higher scores. The common objective is Bradley-Terry style pairwise loss. If output A is preferred over B, you want `r(A) - r(B)` to be positive. The probability of A winning is modeled as `sigmoid(r(A) - r(B))`, then trained with cross-entropy. This setup works well because humans are better at comparing two outputs than assigning absolute quality numbers. But it also means the reward model can learn shortcuts from annotation habits, like overvaluing longer or more agreeable answers. That is why teams combine reward-model metrics with human win rate and safety checks.

RLHF frames reward-model training as pairwise ranking. For each prompt, you sample two completions and collect a preference label that says which completion is better. The reward model outputs a scalar for each completion. Training pushes the chosen completion to a higher score. The standard model is Bradley-Terry: $$P(A \succ B)=\sigma\big(r(A)-r(B)\big)$$ You maximize this probability for observed preferred pairs with log loss. The main practical benefit is label efficiency. Humans can reliably compare outputs even when absolute quality scales are noisy. The trade-off is proxy risk. The reward model can overfit annotator artifacts, such as verbosity or stylistic politeness, instead of true helpfulness. Good pipelines therefore monitor pairwise human win rate and track regressions on capability benchmarks. They also tune the KL anchor during policy optimization so reward gain does not come from drifting into reward-hacking behavior.