Tradeoffs between pairwise LLM-judge comparisons and scalar scoring rubrics

LLM judges can grade in two ways, and the question asks you to weigh them. **Pairwise** evaluation shows the judge two candidate answers and asks which one is better. Scalar scoring shows one answer and asks for an absolute number, like a rating from 1 to 5. Pairwise is more reliable. Judges (like humans) are better at saying which of two things is better than at pinning an exact score. Comparison cancels out a lot of the wobble that comes from a vague rubric. That is its main strength. The weakness of pairwise is cost. To rank many models against each other you need to compare every pair, and that count grows roughly with the square of the number of models. Five candidates already means ten matchups, times the prompts in your eval set. Scalar scoring is cheap: one judge call per output, and you get a number you can plot over time. The catch is that the number drifts. Without clear example answers anchoring what a 3 versus a 5 means, the judge interprets the scale inconsistently from run to run. Use pairwise for a head to head decision between two or three candidates. Use scalar when you need continuous monitoring or are tracking many systems over time.

Two judging modes answer two different jobs. **Pairwise** shows the judge two outputs and asks which is better. It is more reliable because relative comparison is easier than absolute scoring. It resists anchoring, since the judge is not forced to invent a number against a fuzzy scale. The cost is quadratic. Ranking n systems needs roughly n choose 2 comparisons, so cost grows as O(n²). You also get only relative rankings, not a trackable level. Scalar scoring rates each output independently, say 1 to 5. It is cheap, one call per output, and gives an absolute number you can track over time. Its weakness is rubric sensitivity. Without anchored calibration examples, the judge interprets score levels inconsistently and the scale drifts run to run. To turn pairwise wins into a single number, fit a ranking model. Bradley-Terry or Elo converts win-loss records into a latent score per system. Chatbot Arena does exactly this at scale. Choose pairwise for a head to head decision between two or three specific candidates, where reliability matters most. Choose scalar when you need continuous monitoring, want a trend line over time, or must rate many systems where O(n²) is prohibitive.