Writing the LLM-as-judge prompt for an eval pipeline: name three biases to design against and one mitigation each.
Position bias (randomize order), length bias (penalize verbosity in rubric), self-preference bias (cross-family judging). Each distorts scores systematically if unmitigated.
Imagine a cooking contest where the same judge always rates the first dish higher, prefers bigger portions even when the food is the same, and gives extra points to dishes that taste like their own cooking. That is what LLM judges do without mitigation. You fix it by shuffling the tasting order, telling the judge that bigger is not better, and using a judge who cooks differently from the contestants.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LLM-as-judge is the workhorse of modern evaluation pipelines. It scales where human evaluation does not, and it provides rubric-based scoring that execution-based metrics cannot. But LLM judges carry systematic biases that distort scores in predictable directions.
The three most documented biases are position bias (preference for a specific presentation position), length or verbosity bias (preference for longer outputs), and self-preference bias (preference for outputs from the same model family). Each has a concrete mitigation. Understanding all three is necessary because they compound: a single evaluation can be simultaneously affected by all three, pushing the score in the same direction.
Position bias: the order of presentation matters
When a judge evaluates two candidate answers in a pairwise comparison, the order in which they appear influences the score. Most LLM judges show a preference for the first candidate (primacy bias), though some show a recency bias favoring the last candidate.
The effect size is meaningful: on a 100-point scale, position bias can shift scores by 5 to 15 points. For candidates of genuinely similar quality, this is enough to flip the ranking entirely.
The mitigation is straightforward: randomize the presentation order across eval runs. For maximum rigor, evaluate every pairwise comparison in both orderings (AB and BA) and average the scores. The order-sensitivity metric (absolute difference between the two orderings) serves as a diagnostic: high sensitivity means the judge is unreliable on that specific comparison and the result should be flagged.
Randomization doubles eval cost for pairwise comparisons. This is worth it because the alternative is a systematically biased leaderboard.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Chatbot Arena uses randomized presentation order and blind evaluation to control position bias in human and model-based comparisons.
- AlpacaEval 2 implements length-controlled scoring specifically to mitigate verbosity bias, penalizing models that improve scores by generating longer outputs without adding quality.
What an interviewer would ask next. Try answering before peeking at the approach.
QYou implement cross-family judging, but the two families disagree on 30% of comparisons. How do you decide which judge is right?
Neither is right; the disagreement reveals that the comparison is ambiguous. Use a human tiebreaker on the disagreement set. If the disagreement rate is too high for human review, examine whether the rubric is ambiguous on the dimensions where judges disagree. Tighten the rubric and re-run.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Designing the judge prompt without addressing any systematic biases, then trusting the scores as ground truth when they are influenced by presentation order, answer length, and model family matching.
60 second bullets to scan on the way to the call.
Name position bias and its mitigation (order randomization)
Name verbosity bias and its mitigation (rubric penalty or length-controlled scoring)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.