Which statement best describes what each of the three LLM evaluation modes covers?
The three eval modes cover different things: automated metrics for auto-checkable facts, LLM-as-judge for nuanced quality, human review for domain expertise. They are complementary, not interchangeable.
Imagine grading student essays three ways. A spellchecker instantly catches typos and word counts, but it cannot tell if the argument is convincing. A smart teaching assistant reads quickly and judges flow, clarity, and tone, but might miss a subtle factual error in a specialist subject. A senior professor catches the deep domain mistakes, but is slow and expensive, so they only spot-check a few essays. None of the three is enough alone, and they do not do the same job. You run all three because each catches mistakes the others miss. That is exactly how teams evaluate language models: cheap automatic checks, a model judge for quality, and humans for the hard, high-stakes calls.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Evaluating a language model is fundamentally different from grading a classifier. For a classifier you compare a prediction against a label and compute accuracy in one line. For an open-ended LLM output there is usually no single correct string, because quality is multi-dimensional: faithfulness to the source, relevance to the question, completeness, and tone all matter at once. A naive string match against a reference answer fails immediately, because two very different wordings can both be excellent, and one fluent paragraph can be entirely wrong.
This is why the field converged on three complementary evaluation modes rather than one universal metric. Automated metrics check what is mechanically verifiable. LLM-as-judge scores what needs general intelligence but not deep expertise. Human evaluation anchors the high-stakes and domain-specific calls. The single most tested idea for a beginner is that these three are complementary, not competing, and not interchangeable.
This deep dive walks each mode, what it is genuinely good at, where it breaks, and how the three combine into one feedback loop. The unifying lens is simple: for every quality dimension you care about, you pick the cheapest mode that can actually detect its failures, and you accept that no single mode covers the whole space.
Why a single metric cannot work
Classic NLP metrics like BLEU and ROUGE measure word overlap between an output and a reference. They were built for tasks like translation and summarization where the gold answer is fairly constrained. For modern open-ended generation they break badly: a correct answer phrased differently scores low, and a fluent but wrong answer can score high. Embedding-based scores like BERTScore soften this a little, but they still cannot tell whether a claim is true or whether the answer actually solved the user's problem.
The deeper issue is that LLM quality is not one number. An answer can be perfectly fluent yet factually wrong, or factually right yet rambling and off-tone. Faithfulness, relevance, completeness, and style are separate axes, and a system can excel on some while failing others. No single automatic score collapses those axes into one honest verdict.
There is also no canonical reference for most real tasks. A support chatbot, a coding assistant, and a summarizer each have a huge space of acceptable outputs, so 'distance from one gold string' is the wrong frame entirely. What you actually want is to ask, per dimension, whether the output is good enough.
So the practical question becomes: for each quality dimension I care about, what is the cheapest tool that can actually detect its failures? That question is what produces the three-mode split, because different dimensions are detectable by different tools, and forcing them all through one metric is what made early LLM evaluation so unreliable.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- RAGAS combines automated faithfulness scoring with LLM-as-judge calls, while teams still hold out a human-labelled set for calibration.
- OpenAI Evals and Promptfoo run deterministic checks plus judge-based scoring in continuous integration before any model ships.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you decide which eval mode to apply to a given quality dimension?
Ask whether the property is auto-checkable (automated), needs general intelligence but not deep expertise (LLM judge), or requires domain knowledge and high stakes (human). Map each dimension to the cheapest mode that can actually catch its failures.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating the three eval modes as interchangeable. They cover different failure classes, so picking only one leaves a blind spot the others would have caught.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.