What is position bias in pairwise LLM-judge evaluation?

**Position bias** is a flaw in LLM-as-judge evaluation. When you ask a judge model to compare two answers, A and B, side by side and pick the better one, the judge tends to favor whichever answer it sees first (sometimes last) even when the content is identical. Why does this matter? Imagine you always put your new model's answer in slot A and a competitor's in slot B. If the judge leans toward slot A, your model wins more often just because of where it sits, not because it is better. Your win rate is inflated and the comparison is unreliable. The fix is simple and standard. Run every comparison twice, once with A first and once with B first, then average the two results. If an answer only won because of its slot, the swap cancels that out. What survives is the real quality difference. Most modern eval tools like LangSmith, Promptfoo, and RAGAS randomize or swap order by default, so you rarely have to build this yourself. But you should always check that your harness actually does it.

**Position bias** is the tendency of an LLM judge to prefer a candidate based on its position in the prompt, not its quality. In a pairwise comparison, the judge sees answer A then answer B and picks a winner. A biased judge favors the first slot, or sometimes the last, by a few percent, independent of content. Why it is a problem: if one model is always placed first, its win rate is inflated by the slot, not the substance. Your leaderboard then measures position, not quality. That can flip the apparent winner between two close models. The standard fix is order swapping. Run each comparison twice, A-then-B and B-then-A, then average the two outcomes. Any advantage that came purely from position cancels out across the two runs. A related option is single-answer scoring. Rate each answer alone on a 1-to-5 scale, so there is no order to bias. The tradeoff is lower discrimination between close answers. In practice, swap and average is the cheap default mitigation, and good eval frameworks ship it out of the box.