LLM Evaluation
Also known as: LLM eval, Evals
Measuring whether an LLM does what you want, beyond "looks fine to me".
The practice of measuring LLM quality across dimensions like accuracy, fluency, faithfulness, safety, and latency. Combines automated metrics (BLEU, ROUGE, BERTScore) with human evaluation and LLM-as-judge approaches.
In practice
The most under-prepared interview area. Be ready to design a real eval harness (golden set, LLM-as-judge, regression tracking) for a specific product.
Comparisons that include LLM Evaluation
Related topics
Questions that mention this term
- Why is final answer…
- Design an LLM judge prompt with anchor examples and explain how they reduce score variance
- When should you use token level F1 vs exact match in a QA evaluation?
- Is 'unfaithful' the same…
- When is LLM-as-judge an appropriate evaluation method for prompt outputs and what are its known biases that you need to mitigate?
- Clarify the relationship between unfaithfulness and hallucination in RAG systems
Related terms
Chatbot Arena
Crowdsourced blind A/B comparisons of LLM responses, aggregated into an Elo leaderboard.
Eval Harness
A framework for running, scoring, and tracking LLM evals over time; handles golden sets, judges, and regressions.
Golden Set
Curated test cases with expected outputs; your regression suite for prompt and model changes.
Guardrails
Pre- and post-processing layers that block bad inputs/outputs and enforce policy on top of an LLM.
Hallucination
When a model confidently makes up something that isn't true.
HumanEval
Code-generation benchmark: 164 problems with hidden unit tests, scored by whether the generated code passes.