What is position bias in pairwise LLM judge evaluation?
Position bias is an LLM judge's tendency to favor whichever answer appears first (or last), regardless of quality. Fix it by swapping the order and averaging both runs.
Imagine a taste-test where a judge tries two dishes, A then B, and picks a winner. A fair judge would pick the same dish no matter which plate came first. But real judges have a quirk: they often pick whatever they tasted first, just because it came first. So if your dish is always served first, it wins more often, even when it is not actually better. That is position bias in an LLM judge. The cheap fix is simple: run the comparison twice, once with your dish first and once with the rival first, then average the two results. If a dish only wins because of its slot, the swap cancels that out, and you are left with the true quality difference.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Position bias is one of the first failure modes you meet when you use one language model to judge another. The setup is everywhere in 2026: rather than collect slow, expensive human ratings, teams ask a strong judge model like Claude Opus 4.7 or GPT-5.5 to read two candidate answers and say which is better. It is fast and cheap, but the judge is a measurement instrument with its own quirks, and position bias is the most basic one.
The core claim is simple. When a judge compares answer A and answer B in a single prompt, its verdict depends partly on which answer appears first. A fair judge would give the same winner no matter the order. A real judge often leans toward the first slot, or, depending on the prompt, the last.
This deep dive defines the bias precisely, shows why it corrupts win rates, walks the standard fix, and places it next to the other judge biases so you do not over-claim what the fix solves. The thread running through all of it: an LLM judge is not an oracle, it is an instrument, and good evaluation is the discipline of measuring and subtracting the instrument's known errors before you trust the number it prints.
What position bias actually is
In pairwise evaluation you build one prompt that contains both candidate answers, label them A and B, and ask the judge to pick the better one. Position bias is the tendency for that verdict to depend on which answer occupies which slot, rather than only on the content.
Concretely: take two answers, hold them fixed, and run the comparison twice. In the first run, your answer is A and the rival is B. In the second run, you swap them, so the rival is A and yours is B. A perfectly unbiased judge returns the same winner both times. A biased judge sometimes flips its verdict purely because the order changed.
It helps to separate the two ways the bias can point. Some prompt templates produce a primacy effect, where the judge over-weights the first answer it reads, much like a person remembering the opening of a list best. Other templates produce a recency effect, where the judge leans toward the last thing it saw before deciding. Which direction dominates depends on the template wording, the judge model, and even the task category, so you cannot assume one direction in advance.
Studies on judge models put the effect in the range of a few percent up to around ten percent of pairs, depending on the prompt template and the judge. It is small per pair but consistent, which is exactly what makes it dangerous: a small, systematic tilt repeated over thousands of comparisons moves an aggregate win rate in a fixed direction. Random noise would average out; a systematic tilt does not, and that is the difference between a harmless wobble and a corrupted leaderboard.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangSmith pairwise evaluators randomize candidate order by default and run at temperature zero for reproducible verdicts.
- Promptfoo surfaces position-swapped pairwise comparisons so teams can see verdicts flip when the order changes.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you measure how strong the position bias is in your specific judge?
Run the same pair in both orders across a sample, count how often the verdict flips, and report the flip rate as a bias estimate. A high flip rate means the slot, not quality, is driving verdicts.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Running pairwise judging with a fixed order and trusting the win rate. The favored slot inflates one model's score, so the result reflects position, not quality.
60 second bullets to scan on the way to the call.
Define position bias in one plain sentence
Explain why a fixed order inflates one model's win rate
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.