Statistics

PremiumYour eval shows Model…

Explain the sign reversal failure mode in LLM judge calibration and how to prevent it

PremiumWhat is the sign…

How do you determine if a 2% eval score drop between model versions is statistically significant?

Which test is most appropriate for determining if a binary metric LLM eval drop is statistically significant?

Explain the pass@k metric and why the unbiased estimator matters for code gen evaluation

What does Cohen's kappa measure in human LLM evaluation?

How large should a golden eval set be, and what signals tell you when you have enough examples?

When should you use Fleiss's kappa instead of Cohen's kappa in a human eval study?

How do you measure whether an LLM judge is well calibrated against human raters?

Complete: Cohen's kappa >___ is 'substantial' agreement; >___ is 'almost perfect' agreement