LLM As Judge Calibration

Your LLM judge gives everything a 4 out of 5. Your colleague says the judge is not calibrated. What does calibration mean here?

Explain the sign-reversal failure mode in LLM judge calibration and how to prevent it

What is the sign-reversal failure mode in LLM-as-judge calibration?

How do you measure whether an LLM judge is well-calibrated against human raters?

Complete: Cohen's kappa >___ is 'substantial' agreement; >___ is 'almost perfect' agreement