Position bias is the tendency of LLM judges to prefer whichever candidate appears first (or last) in the prompt, regardless of actual quality. This ordering artifact systematically inflates the win rate of one model when it is always placed first, making evaluation results unreliable unless the ordering is randomized or swapped.
**Position bias** is a flaw in LLM-as-judge evaluation. When you ask a judge model to compare two answers, A and B, side by side and pick the better one, the judge tends to favor whichever answer it sees first (sometimes last) even when the content is identical.
Why does this matter? Imagine you always put your new model's answer in slot A and a competitor's in slot B. If the judge leans toward slot A, your model wins more often just because of where it sits, not because it is better. Your win rate is inflated and the comparison is unreliable.
The fix is simple and standard. Run every comparison twice, once with A first and once with B first, then average the two results. If an answer only won because of its slot, the swap cancels that out. What survives is the real quality difference.
Most modern eval tools like LangSmith, Promptfoo, and RAGAS randomize or swap order by default, so you rarely have to build this yourself. But you should always check that your harness actually does it.
**Position bias** is the tendency of an LLM judge to prefer a candidate based on its position in the prompt, not its quality. In a pairwise comparison, the judge sees answer A then answer B and picks a winner. A biased judge favors the first slot, or sometimes the last, by a few percent, independent of content.
Why it is a problem: if one model is always placed first, its win rate is inflated by the slot, not the substance. Your leaderboard then measures position, not quality. That can flip the apparent winner between two close models.
The standard fix is order swapping. Run each comparison twice, A-then-B and B-then-A, then average the two outcomes. Any advantage that came purely from position cancels out across the two runs.
A related option is single-answer scoring. Rate each answer alone on a 1-to-5 scale, so there is no order to bias. The tradeoff is lower discrimination between close answers. In practice, swap and average is the cheap default mitigation, and good eval frameworks ship it out of the box.
Quick reference
TL;DR
Position bias is an LLM judge's tendency to favor whichever answer appears first (or last), regardless of quality. Fix it by swapping the order and averaging both runs.
Imagine a taste-test where a judge tries two dishes, A then B, and picks a winner. A fair judge would pick the same dish no matter which plate came first. But real judges have a quirk: they often pick whatever they tasted first, just because it came first. So if your dish is always served first, it wins more often, even when it is not actually better. That is position bias in an LLM judge. The cheap fix is simple: run the comparison twice, once with your dish first and once with the rival first, then average the two results. If a dish only wins because of its slot, the swap cancels that out, and you are left with the true quality difference.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tasks with a cheap, exact gold answer such as multiple choice or structured extraction Exact match against the gold label is unbiased and far cheaper, so pairwise judging adds bias for no benefit.
Evaluating a single system in isolation rather than comparing two candidates There is no pair and no order, so position bias does not apply; use scalar rubric scoring instead.
Extremely high-volume evaluation where the 2x swap cost is prohibitive Doubling judge calls per pair can dominate the bill, so teams sometimes sample or switch to scalar scoring.
Quick scan · 2–4 min
Everything important, quickly.
3 min: define position bias, explain why a fixed order inflates win rate, give the swap and average fix, and contrast with single-answer scalar scoring.
Real-world examples
Real products, models, and research that use this idea.
1LangSmith pairwise evaluators randomize candidate order by default and run at temperature zero for reproducible verdicts.
2Promptfoo surfaces position-swapped pairwise comparisons so teams can see verdicts flip when the order changes.
3Chatbot Arena and MT-Bench studies documented position bias as a core reason to randomize and swap answer order.
4RAGAS and TruLens lean on grounded, order-controlled judging rather than raw fixed-order pairwise scores.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you measure how strong the position bias is in your specific judge?
Sign in or go Pro to see the approach for this follow-up.
QWhy does swapping and averaging not fix self-preference or length bias too?
Sign in or go Pro to see the approach for this follow-up.
QWhen would single-answer scalar scoring be a better choice than swapped pairwise?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Running pairwise judging with a fixed order and trusting the win rate. The favored slot inflates one model's score, so the result reflects position, not quality.