Writing the LLM-as-judge prompt for an eval pipeline: name three biases to design against and one mitigation each.

LLM judges have three well-documented biases that distort scores if you do not design against them. **Position bias.** The judge tends to prefer whichever answer appears first (or last, model-dependent). Fix: randomize the order across eval runs and average scores from both orderings. **Length bias.** The judge scores longer answers higher even when the extra words add nothing. Fix: add a rubric instruction that explicitly penalizes unnecessary verbosity, or set a target length range and deduct for exceeding it. **Self-preference bias.** The judge prefers outputs that match its own style. Claude rates Claude higher; GPT rates GPT higher. Fix: use a judge from a different model family than the model being evaluated. All three biases can compound in a single comparison. If one answer is from the same family as the judge, is longer, and appears first, all three biases push the score in the same direction.

**Position bias.** Judges prefer whichever candidate appears first (or last). Mitigation: randomize candidate order across runs. For pairwise comparisons, evaluate both orderings and average. If the score flips with order, position bias is active and both orderings are mandatory. **Length or verbosity bias.** Judges score longer answers higher regardless of whether the extra length adds substance. Mitigation: add an explicit rubric instruction penalizing unnecessary verbosity. Alternatively, use length-controlled scoring with a target word range; the judge deducts for exceeding the range without adding content. **Self-preference bias.** A judge model favors outputs that match its own generation style. Mitigation: use a judge from a different model family than the candidate. Evaluate Claude outputs with GPT-5.5; evaluate GPT outputs with Claude Opus 4.7. Cross-family judging breaks the style-matching loop. These biases compound. In a pairwise eval where one candidate is from the judge's family, is longer, and appears first, the effect can shift scores 10 to 20 points on a 100-point scale. Enough to flip rankings between genuinely similar candidates.