Position bias, self-preference bias, and weakness on factual grounding are real concerns. Determinism and equal cost are not, judges sample at T greater than 0 by default and usually cost more than the candidate.
Imagine asking a famous chef to judge a cooking competition. Three things go wrong even before the food arrives. First, they remember the first dish best (position bias). Second, they unconsciously prefer the cooking style closest to their own training (self-preference). Third, they are great at saying which dish LOOKS prettier but cannot easily tell whether the chef used the right amount of salt unless they taste-test against the recipe (weak on factual grounding). Two things people sometimes worry about are not really problems: the chef does not always score the same plate identically (unless you pin them down), and the chef is usually not cheaper than the cooks they are judging.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LLM-as-judge is the dominant evaluation pattern in 2026 because it scales: a single Claude Opus 4.7 call can score thousands of candidate outputs per hour at a fraction of human-evaluator cost. The catch is that the judge is itself a measurement instrument with known systematic error, and a serious eval engineer learns to recognise and correct that error.
This question hits five common claims about LLM judges, three of which are real concerns and two of which are misconceptions. The deep dive walks each bias, the standard mitigation, the conditions under which the mitigation works, and the production stacks that have codified these patterns.
Position bias: the easiest one to fix
When asked to compare two candidate answers A and B side by side, an LLM judge does not weight content evenly. Across multiple studies the judge prefers position 1 (or sometimes position 2, depending on the prompt template) by 3 to 10 percent independent of content. The bias is consistent enough that you can detect it just by running A vs B and then B vs A and watching the scores diverge.
The fix is mechanical. For every pairwise comparison, run both orders and average. Most modern eval frameworks (LangSmith, Promptfoo, RAGAS) ship this as default behavior. The cost is a 2x judge call count, well worth it for stable rankings.
An alternative is to skip pairwise entirely and use single-answer rating (1 to 5 scale, or 0 to 1 score). No order to bias on. The downside: lower discrimination power for small quality differences, since the judge tends to anchor on a few standard scores.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- RAGAS uses Claude Opus 4.7 or GPT-5.5 as judge but enforces claim-level entailment for faithfulness, not holistic judgment.
- LangSmith Evaluators ship position-randomised pairwise comparisons as the default with T=0.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you measure whether your LLM judge actually agrees with human labels?
Collect a small held-out human-labelled set (200 to 500 examples), compute Cohen's kappa or Spearman correlation between judge and human scores. Quarterly recalibration is the production norm because model versions drift.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating an LLM judge as a calibrated oracle. Judges have measurable biases (position, self-preference) and weak factual grounding, all of which corrupt the eval if uncontrolled.
60 second bullets to scan on the way to the call.
Position bias and the order-swap mitigation
Self-preference bias across model families
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.