Why do LLM judges tend to rate longer answers higher, even when verbosity reduces quality?

**Length bias** is the tendency of an LLM judge to score longer answers higher, even when the extra words add nothing or actively hurt quality. The correct option pins down where it comes from: the judge learned it during RLHF training. Here is the chain. To align a model, you collect human preference data: people compare two answers and pick the better one. Crowd annotators, rating quickly, often pick the longer, more detailed-looking response because it signals effort and coverage. That preference gets baked into the reward model and, downstream, into any model used as a judge. So the judge does not reason 'this is longer, therefore better'. It simply inherited a statistical habit: length correlates with the 'good' label in its training. The result is a systematic thumb on the scale toward verbosity. The other options are wrong. Longer answers do not reliably contain more correct facts; padding is common. Perplexity and tokenization are unrelated to this preference. And sliding-window attention is an unrelated architecture detail, not the cause. The real story is human-preference data, not model plumbing.

**Length bias** in an LLM judge is a learned artifact, not a rational signal. The correct answer traces it to RLHF. When you align a model, you gather human preference labels. Annotators comparing two answers tend to pick the longer, more detailed one, because length signals effort. That preference is encoded into the reward model. Any model later used as a judge carries the same habit. So the judge associates length with the 'good' label it was trained on. A padded answer beats a concise correct one. This inflates scores for models that optimize verbosity over accuracy. Why the distractors fail. Longer answers do not reliably hold more correct facts. Verbosity often adds filler or hedging. Perplexity and tokenization play no role here. Sliding-window attention is an architecture detail, not a scoring cause. The practical takeaway is detection plus mitigation. You detect it with a length-controlled win rate, comparing systems while conditioning on response length. You mitigate it with an explicit rubric that scores correctness and concision as named axes. You can also penalize length directly. AlpacaEval 2.0 added length control because raw win rate rewarded padding.