Per-comparison randomization of which side each response appears on converts position bias from a systematic distortion into noise that averages out across the dataset.
Imagine two pies in a contest and the judges stand on one side of the room. If pie A is always on the left, the left-side judges naturally lean toward it. The contest is biased in pie A's favor before anyone tastes anything. The fix is to randomly swap which pie sits on which table for every judge. The favored side still gets picked too often, but now half the time it is pie A and half the time it is pie B, so the bias becomes noise that cancels out. Showing the judges the scores from a previous contest would defeat the purpose. Fixing the pie locations is exactly the bug. Skipping the hard comparisons just throws out the most useful data.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Position bias is one of the most-studied and most-underestimated sources of contamination in preference-data collection. It is small per comparison, often only 5 to 15 percent excess preference for one side, but it accumulates systematically over thousands of labels and propagates from labelers to the reward model to the policy. A well-designed labeling UI breaks the systematic part of the bias by introducing per-comparison randomness.
This question targets a common UI design mistake: fixing the candidate model to a consistent slot for rater convenience. That convenience costs you a contaminated reward model, which is exactly the artifact downstream RL will exploit.
This deep dive walks through why position bias is structural in both human and LLM judging, why per-comparison randomization works statistically, what the audit discipline looks like in production, and why each of the three distractors represents a recognizable real-world mistake.
Why position bias is structural
Position bias has multiple sources that all push in the same direction. Cognitive anchoring causes humans to weight the first option encountered more heavily. Left to right reading cultures process the left-side response first by default. Mouse-cursor proximity to the left option (in most layouts) reduces the friction of selecting it. Visual hierarchy conventions place the more prominent option on the left or top.
The net effect across studies is a 5 to 15 percent excess preference for the left or first response when quality is genuinely equal. The effect is robust across raters, languages, and content types, though its magnitude varies.
LLM judges show the same bias. Models trained on web text have absorbed the human pattern. Even when explicitly told to avoid position bias in the system prompt, judges measurably favor the first response in side by side comparisons. The AlpacaEval and MT-Bench teams documented this empirically and built their evaluation protocols around it.
The per-comparison bias is small, but it compounds. A reward model trained on 100,000 labels with a 10 percent systematic position bias on the candidate model side learns to favor the candidate by an amount roughly proportional to that bias. The policy trained against that RM then exploits the bias, producing responses that are not actually better, just structured in ways the biased RM prefers.
Position bias is structural because human attention has anchoring effects: the first option gets read more carefully, the second is compared against the first. Studies on rater behavior in pairwise tasks (Zheng et al. 2024 on MT-Bench, the Anthropic 2024 annotator behavior writeup, and academic work on judgment under uncertainty going back to Tversky and Kahneman) consistently show 2 to 8 percent preference for the first-listed option even when content is balanced. Without a mitigation, the entire dataset encodes a bias toward whatever role (model A vs model B, candidate vs baseline) is structurally listed first.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- AlpacaEval evaluates LLM judges by running both orderings and averaging, the standard practice to neutralize position bias in LLM-as-judge
- Scale AI's preference-labeling UI randomizes display position per-comparison and logs the assignment for downstream auditing
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you audit residual position bias after randomization is in place?
Compute the marginal win rate by displayed position over a holdout slice; under correct randomization it should be 50 percent within statistical noise. Systematic deviation indicates randomization is broken or a secondary bias (length, formatting) is acting through position.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Fixing the candidate model to a consistent slot for rater convenience. That is exactly what creates systematic position-bias contamination of every label.
60 second bullets to scan on the way to the call.
Why position bias is structural in both human and LLM judging
How per-comparison randomization converts systematic bias into noise
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.