Eval Fundamentals

Pick the best definition: in LLM evaluation, a metric is _____.

A teammate asks what 'running evals' means in the LLM context. How would an engineer explain it without jargon?

Your LLM judge gives everything a 4 out of 5. Your colleague says the judge is not calibrated. What does calibration mean here?

You have no reference answers for your eval set. Does that mean you cannot evaluate the model?

Explain the difference between pointwise and pairwise evaluation in one breath.

Offline eval now or wait for the A/B test results? Pick the key difference between online and offline evaluation.

Automated eval vs human eval: which is better for catching subtle factual errors in medical text?

Explain why string-match accuracy is systematically misleading for LLM evaluation

Why string-match accuracy fails as an LLM evaluation metric

Which properties correctly describe the complementary roles of the three LLM evaluation modes?

Which statement best describes what each of the three LLM evaluation modes covers?

Why is reporting only mean eval score dangerous for LLM quality tracking?

Explain context recall in RAG evaluation and why it is more expensive than context precision

What is the practical guidance on golden eval set size for LLM systems?