Pick when pairwise judging beats scalar judging in an eval workflow
Pairwise judging wins for A/B comparisons because it dodges calibration drift; scalar judging wins for tracking absolute quality over time across heterogeneous prompts.
Imagine asking a wine taster two different questions. Question one: which of these two glasses tastes better? Even a tipsy taster will pick the better one almost every time, because the comparison is in front of them. Question two: rate this single glass from one to ten. Now the taster has to remember what a six tasted like last week and what a six should mean today. Their answer drifts. The first question is pairwise judging. The second is scalar judging. For deciding whether glass A or glass B should ship to customers, the comparison is cleaner. For tracking 'are our wines getting better over the year', you need the rating because you cannot run every glass head to head against every other glass.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Pairwise versus scalar is one of those eval-design choices that looks like a preference but is actually a constraint problem. Each method answers a different question, has a different variance profile, and a different bias surface. The right answer in any specific case depends on whether you have a natural opponent and whether you need an absolute number or a relative verdict.
This card walks through the variance argument first, then the two domains where each method is unavoidable, then the bias surfaces specific to each, and closes with the production pattern that uses both.
The variance argument
An LLM judge giving an absolute score on a 1-5 scale has to do two hard things at once. It has to assess the output and it has to anchor the assessment against an internal sense of what each number means. The second step is where most of the noise comes in. The judge's sense of '4' shifts with the first few examples it sees, with the prompt context, and with the underlying model version.
Why pairwise dodges this
A pairwise judge is told 'pick the better one'. That is a 2-way decision with a clean noise floor at 50%. The judge does not need to remember what a 4 felt like last week; it just needs the relative ordering for the two outputs in front of it. Empirical studies on LLM-as-judge consistently show pairwise agreement with human labels running 10-20 points higher than scalar agreement on the same task and judge model.
What this buys in practice
A prompt-change A/B with 200 examples judged pairwise will reach statistical significance faster than the same 200 examples judged scalar. The pairwise variance is lower, so the confidence interval on the win rate is tighter. For change-decision workflows, that translates directly into faster ship cycles.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | Pairwise (A vs B) | Scalar (1-5) |
|---|---|---|
| Variance | Low; binary choice | Higher; calibration drift |
| Best use | A/B prompt or model changes | Trend dashboards over time |
| Needs opponent | Yes | No |
| Common bias | Position, verbosity, self-preference | Anchoring, score compression |
| Aggregation | Bradley-Terry / Elo | Mean / percentile |
Real products, models, and research that use this idea.
- LMSYS Chatbot Arena ranks frontier models with pairwise human votes aggregated into a Bradley-Terry Elo score.
- Braintrust ships both Pairwise and Scorer abstractions as first-class objects on a trace.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you neutralize position bias in pairwise judging?
Randomize A/B order per example, or run both orderings and only count consistent verdicts; throw away ties.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating pairwise and scalar as interchangeable; they answer different questions and a workflow needs both at different stages.
60 second bullets to scan on the way to the call.
Why pairwise has lower variance than scalar
Calibration drift and anchoring in scalar scoring
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.