Writing the LLM-as-judge prompt for an eval pipeline: name three biases to design against and one mitigation each. · Zenaique

Attempt it

Writing the LLM-as-judge prompt for an eval pipeline: name three biases to design against and one mitigation each.

Free · 2 AI evals / day

Quick reference

TL;DR

Position bias (randomize order), length bias (penalize verbosity in rubric), self-preference bias (cross-family judging). Each distorts scores systematically if unmitigated.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a cooking contest where the same judge always rates the first dish higher, prefers bigger portions even when the food is the same, and gives extra points to dishes that taste like their own cooking. That is what LLM judges do without mitigation. You fix it by shuffling the tasting order, telling the judge that bigger is not better, and using a judge who cooks differently from the contestants.

Key concepts

LLM-as-Judge Position Bias Self-preference Bias

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Open with the framing that judge biases are systematic, not random. Walk the three biases in order: position (randomize order, run both orderings), verbosity (rubric penalty, length-controlled scoring), self-preference (cross-family judging). Explain how all three compound in a single comparison. Close with the production discipline: measure bias diagnostics as part of the eval pipeline before trusting scores.

Real-world examples

Real products, models, and research that use this idea.

Chatbot Arena uses randomized presentation order and blind evaluation to control position bias in human and model-based comparisons.
AlpacaEval 2 implements length-controlled scoring specifically to mitigate verbosity bias, penalizing models that improve scores by generating longer outputs without adding quality.
Anthropic's internal evaluation pipelines use cross-family judging: Claude outputs are judged by non-Claude models, and vice versa, to control self-preference bias.
The LMSYS leaderboard reports both raw and length-adjusted scores, making verbosity bias visible to researchers comparing model rankings.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QYou implement cross-family judging, but the two families disagree on 30% of comparisons. How do you decide which judge is right?

Sign in or go Pro to see the approach for this follow-up.

QA fourth bias: authority bias, where the judge defers to outputs that cite sources or use confident language. How would you test for it?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Designing the judge prompt without addressing any systematic biases, then trusting the scores as ground truth when they are influenced by presentation order, answer length, and model family matching.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.