LLM Evaluation
Also known as: LLM eval, Evals
Measuring whether an LLM does what you want, beyond "looks fine to me".
The practice of measuring LLM quality across dimensions like accuracy, fluency, faithfulness, safety, and latency. Combines automated metrics (BLEU, ROUGE, BERTScore) with human evaluation and LLM-as-judge approaches.
In practice
The most under-prepared interview area. Be ready to design a real eval harness (golden set, LLM-as-judge, regression tracking) for a specific product.
Comparisons that include LLM Evaluation
Related topics
Questions that mention this term
- Why string match accuracy fails as an LLM evaluation metric
- When should automated eval escalate to human review in a production LLM product?
- When is self-refine (LLM…
- Running LLM-as-judge on your…
- The model provider just updated the model behind the API endpoint. Describe the regression testing process before allowing traffic.
- You suspect your custom…
Related terms
Chatbot Arena
Crowdsourced blind A/B comparisons of LLM responses, aggregated into an Elo leaderboard.
Eval Harness
A framework for running, scoring, and tracking LLM evals over time; handles golden sets, judges, and regressions.
Golden Set
Curated test cases with expected outputs; your regression suite for prompt and model changes.
Guardrails
Pre- and post-processing layers that block bad inputs/outputs and enforce policy on top of an LLM.
Hallucination
When a model confidently makes up something that isn't true.
HumanEval
Code-generation benchmark: 164 problems with hidden unit tests, scored by whether the generated code passes.