What is the standard mitigation for position bias in pairwise LLM-judge evaluation?

**Position bias** is a quirk of LLM judges: when you show a judge two answers and ask which is better, it tends to favor whichever appears first (sometimes second), regardless of actual quality. The effect is real and measurable, often shifting verdicts by several percent. The correct answer is option A. You run every comparison twice, once as A versus B and once as B versus A, then either keep only the verdicts that stay consistent across the swap or average the two outcomes. A verdict that survives the swap is far more likely to reflect quality than position. Why not the others? Option B (always put the challenger first) does not remove the bias, it just relocates it onto a fixed slot, so your score is still tilted. Option C (use a scalar rubric) does cut position effects since there is no first or second, but it brings its own anchoring artifacts and is not what people mean by the standard mitigation. Option D (bigger judge) is a myth. Position bias is a systematic behavioral pattern that persists even in the strongest frontier judges. Size does not buy it away.

The correct answer is option A. Position bias means an LLM judge favors whichever candidate sits in a given slot, independent of content. The standard fix is the order swap. Run every pairwise comparison twice: once as A versus B, once as B versus A. Then either keep only the verdicts that agree across both orders, or average the two outcomes. A verdict that survives the swap is evidence of a genuine quality preference. A flip is a caught instance of position bias, so you discard it or down-weight it. This matters because the swap is also your detector. The consistency rate across orders is a direct, auditable measure of how biased your judge is on this task. The distractors each fail differently. Option B just moves the bias to a fixed position, so the score stays tilted. Option C reduces position effects but trades them for scale-anchoring artifacts, and it is not the canonical pairwise fix. Option D is wrong because bias is behavioral, not capacity-limited. Even top frontier judges flip on swap.