What is the sign-reversal failure mode in LLM-as-judge calibration?

**Sign reversal** is a calibration failure mode in LLM-as-judge eval where a correction meant to remove bias actually flips the true quality ranking, so a worse model ends up scoring higher. Here is the setup. An LLM judge has systematic biases, since it favors certain styles. To compare two models fairly, you try to subtract that bias out. The danger: if you estimate the bias in the wrong direction, you do not just fail to fix it, you double it. The genuinely better model gets pushed down and the worse one gets pushed up. The nasty part is the confidence. After correction the numbers can look clean, with a tight confidence interval saying the worse model wins. So you ship the wrong model believing the data told you to. This is closely related to Simpson's paradox: an aggregate number can move one way while every underlying slice moves the other way. The aggregate score can rise even when the model got worse on every query type, simply because the mix of queries shifted. The fixes are the same in spirit: never trust a single aggregate score, report metrics per slice, and validate the correction against human labels before believing it.

**Sign reversal** is the calibration failure where a cross-model bias correction is applied in the wrong direction, inverting the true ranking. A genuinely better model is reported as worse, often with apparent statistical confidence. The mechanism is shared bias. Suppose Model A is actually better but shares a stylistic bias with the judge that Model B lacks. A naive correction assumes the wrong model is the reference. It then subtracts the bias from A and effectively adds it to B. The correction does not shrink error. It compounds it and flips the sign. The deeper root is Simpson's paradox. An aggregate score can improve while every slice degrades, purely from mix shift. Per-query-type accuracy can drop everywhere, yet the mean rises because easy queries grew as a fraction of the set. It is acute in item response theory calibration. IRT estimates item difficulty jointly across models. If the joint fit anchors on the wrong model as stronger, the shared difficulty parameters carry that error into every ability estimate. The bug hides inside a principled-looking model. The defense is structural. Stratify and report per slice. Hold the slice mix fixed across versions. Validate any correction against human labels before trusting it.