Statistics

Your eval shows Model B scores 82.1% vs Model A at 80.5% on 200 test cases. Your manager wants to ship Model B immediately. What question should you ask first?

Explain the sign-reversal failure mode in LLM judge calibration and how to prevent it

What is the sign-reversal failure mode in LLM-as-judge calibration?

How do you determine if a 2% eval score drop between model versions is statistically significant?

Which test is most appropriate for determining if a binary-metric LLM eval drop is statistically significant?

Explain the pass@k metric and why the unbiased estimator matters for code gen evaluation

What does Cohen's kappa measure in human LLM evaluation?

How large should a golden eval set be, and what signals tell you when you have enough examples?

When should you use Fleiss's kappa instead of Cohen's kappa in a human eval study?

How do you measure whether an LLM judge is well-calibrated against human raters?

Complete: Cohen's kappa >___ is 'substantial' agreement; >___ is 'almost perfect' agreement