Explain the sign-reversal failure mode in LLM judge calibration and how to prevent it

**Sign reversal** is when your eval reports that Model B beats Model A overall, even though Model A actually scores higher on every individual category of question. The aggregate number points the wrong way. This is Simpson's paradox. It happens when the two models are not graded on the same mix of questions. Suppose Model A mostly gets easy questions and Model B mostly gets hard ones, or the slices have very different sizes. The pooled average then mixes together apples and oranges, and the mix itself drives the result more than the actual quality. The danger is that the pooled number looks clean and authoritative. A dashboard shows one score per model, with a confidence interval, and it confidently tells you the wrong winner. The fix is to always stratify. Break the eval into meaningful slices (easy or hard, short or long, by language or topic), compute the score per slice, and only then combine using one fixed weighting that is the same for both models. If a model wins every slice but loses the pooled average, trust the slices.

**Sign reversal** is when the aggregate eval declares the worse model the winner. Model A beats Model B inside every slice, yet B wins the pooled mean. This is Simpson's paradox applied to LLM evaluation. The cause is a mix shift between slices. The two models are not scored on the same distribution of question types. If A's results are dominated by a hard slice where everyone scores low, A's pooled mean drops, even though A leads within that slice. The pooled mean is a weighted average where the weights are each model's own slice sizes. The fix is to fix those weights. Pick one reference mix and apply it to both: $$\bar{s}_M = \sum_k w_k \, s_{M,k}, \qquad \sum_k w_k = 1$$ Shared weights make the paradox impossible. Here `w_k` is one slice weight, identical across models, not each model's own slice count. Detection is simple. Always stratify. Report per-slice scores alongside any aggregate. If a model wins every slice but loses the pooled number, you have a mix shift, and the slices are the truth. Prevent it by drawing balanced eval sets and reporting mix-weighted aggregates by default.