How do you detect and mitigate length bias when using an LLM judge in a production eval pipeline?

**Length bias** is the tendency of an LLM judge to score longer answers higher even when the extra length adds no real quality. The judge learns that thorough answers tend to be good, then over-generalises and starts rewarding verbosity itself. To detect it, run an experiment that isolates length. Build a small set of answer pairs where one is short and correct, and the other says the same correct thing but is padded with filler. Humans agree the short one is at least as good. If the judge keeps picking the padded answer, the bias is confirmed. A cheaper signal is to correlate judge scores against response length across your normal eval set. A strong positive correlation with no matching jump in real quality is a red flag. To fix it, you stack several mitigations. Tell the judge explicitly in its prompt not to reward length. Add anchor examples to the rubric that show a short answer winning. Use a panel of judges from different model families and average them. Adjust scores statistically for length after the fact. Finally, monitor the score versus length correlation in production over time, not just once, because the bias drifts as you change models and prompts.

**Length bias** means the judge rewards verbosity, not quality. Detection comes first. Measure it. Build a diagnostic set of matched pairs. One answer is short and correct. The other adds the same content plus filler. Humans agree the short answer is at least as good. Run the judge pairwise and count how often it picks the padded answer. As a cheaper signal, correlate judge scores against token length across your live eval set. Strong positive correlation with no real quality gain confirms bias. Then layer mitigations across three levels. At the prompt level, instruct the judge to ignore length and grade substance. At the data level, add rubric anchor examples where a short answer wins. At the process level, run a panel of judges from different families and average them. Length-controlled win rate is the Arena-style fix. You regress the win signal on a length-difference feature. Then you report the win rate at zero length difference. This removes the length effect statistically without altering the answers. Finally, monitor the score versus length correlation as a production metric and alert on drift. Do not just truncate every answer to a fixed length. Truncation destroys real content and hides the bias instead of measuring it.