Why do reward models often reward longer responses, and how do you mitigate it?

A reward model is a scorer trained on human choices between two answers. **Length bias** means it gives longer answers higher scores even when they are not better. It starts with the people who label the data. A longer answer often looks more thorough, so raters pick it a bit more often than chance. In one study, always picking the longer answer matched the human choice between 55.7% and 63.1% of the time, depending on the dataset. The reward model learns that length predicts winning. Then RLHF (training the model to raise the reward score) makes it worse. Adding words is an easy way to raise the score, so the model learns to pad its answers. You can reduce it in three places: - Data: balance the pairs so the longer answer does not usually win. - Reward: subtract a small penalty for extra length, or train the scorer to ignore length. - Evaluation: compare answers at similar length, and watch average length as training goes on.

Reward models show **length bias** because their training labels do. Human and LLM raters often read length as thoroughness, so in common preference datasets the longer response wins more often than chance. The reward model learns length as a cheap feature. During RL the policy exploits it, because adding tokens raises reward faster than improving quality does. One study found that a reward based only on length reproduced most of the RLHF gains over SFT. Mitigations work at three points. In the data, balance pairs by length so that length stops predicting the label. In the reward, subtract a length penalty, or train a reward model with a separate length head and drop that head during RL, as ODIN does. For DPO, add a length regularizer or use a length-normalized reward such as SimPO's average log-probability. Then check with evaluation. Report length-controlled win rates and track mean length per checkpoint. If the gain disappears at matched length, it was length, not quality.