What is the primary scalability limitation of pairwise LLM-judge evaluation compared to scalar scoring?

An **LLM judge** can score outputs two ways. Pairwise puts two answers side by side and asks which is better. Scalar (absolute) scoring rates one answer alone on a fixed scale, like 1 to 5. Pairwise is the more reliable method. Humans and LLMs are both better at relative judgments than absolute ones. It is easier to say 'A is better than B' than to commit to 'A is a 7 out of 10.' The catch is cost. To fully rank n models pairwise, you compare every pair, which is C(n,2) = n(n-1)/2 comparisons. At 10 models that is 45 comparisons; at 20 it jumps to 190. That quadratic growth is the scalability limit, and it is why the correct option is the O(n²) one. Scalar scoring sidesteps this. You score each model once, independently, so cost grows linearly with the number of models. That makes it ideal for continuous tracking, like a dashboard that re-scores every build. The trade-off: scalar scores suffer anchoring and scale drift. Without a side by side reference, the judge's notion of '4 out of 5' wanders across runs.

Pairwise and scalar are the two LLM-judge scoring modes. Pairwise compares two outputs and picks a winner. Scalar rates one output alone on a fixed scale. **Reliability.** Pairwise wins here. Relative judgments are easier and more consistent than absolute ones. Judges agree with humans more often when choosing between two answers than when assigning a number to one. The scalability catch is the answer to the question. Ranking n models pairwise needs every pair compared. That count is: $$\binom{n}{2} = \frac{n(n-1)}{2} = O(n^2)$$ At 10 models that is 45 comparisons; at 20 it is 190. Scalar scoring is O(n): score each model once, independently. So scalar is the cheap, continuous-tracking choice. Pairwise is the reliable but quadratic one. **Scalar's weakness.** Absolute scores drift. Without a side by side anchor, the judge's '4 out of 5' wanders between runs, prompts, and rubric phrasings. This is anchoring sensitivity and scale drift. The wrong options. Pairwise is reference-free, not reference-required. It is more reliable, not less. It does convert to a leaderboard via ELO or Bradley-Terry. The only true limitation is the quadratic comparison count.