Evaluating a language model is harder than grading a math quiz. There is rarely one correct string to match against, because quality is multi-dimensional: is the answer faithful, relevant, complete, and in the right tone? No single tool measures all of that, so modern teams use **three complementary modes**.
First, automated metrics. These are cheap, fast, and deterministic checks for things with a clear right answer: did the code run, is the JSON valid, does the output match the gold label? Great for what is auto-checkable, useless for nuance.
Second, LLM-as-judge. Here a strong model scores another model's output on fuzzy qualities like helpfulness or clarity. It scales to thousands of examples cheaply and handles judgments that need general intelligence, though it carries known biases.
Third, human evaluation. Slow and expensive, so you only spot-check, but humans are the ground truth for high-stakes or domain-specific calls a model cannot reliably make.
The key idea: they are complementary, not competing. Each catches failures the others miss, so good teams run all three rather than betting on one.
The correct framing is that the three modes have **complementary coverage**, not interchangeable coverage.
Automated metrics handle auto-checkable properties cheaply and deterministically. Examples include code execution, exact match on a gold label, schema validation, and string overlap scores. They are fast and unbiased, but blind to nuance.
LLM-as-judge handles the fuzzy middle. A strong model scores qualities like helpfulness, coherence, or tone that resist hard coding. It scales to thousands of examples for cents each. The tradeoff is known bias: position bias, self-preference, and weak factual grounding.
Human evaluation is the ground-truth anchor. It is slow and costly, so you only spot-check, but humans catch domain-specific and high-stakes errors a model misses. Human labels also calibrate whether your judge is trustworthy.
Why not just pick one? Because each mode catches a different failure class. An automated metric cannot grade tone. A judge cannot reliably verify niche medical facts. A human cannot review a million outputs. Running all three is defense in depth, not redundancy. The wrong options confuse this. The modes are not interchangeable, human review does not scale, and a judge should usually not be the same family as the model it grades.
Quick reference
TL;DR
The three eval modes cover different things: automated metrics for auto-checkable facts, LLM-as-judge for nuanced quality, human review for domain expertise. They are complementary, not interchangeable.
Imagine grading student essays three ways. A spellchecker instantly catches typos and word counts, but it cannot tell if the argument is convincing. A smart teaching assistant reads quickly and judges flow, clarity, and tone, but might miss a subtle factual error in a specialist subject. A senior professor catches the deep domain mistakes, but is slow and expensive, so they only spot-check a few essays. None of the three is enough alone, and they do not do the same job. You run all three because each catches mistakes the others miss. That is exactly how teams evaluate language models: cheap automatic checks, a model judge for quality, and humans for the hard, high-stakes calls.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tasks with a cheap, exact gold answer like multiple choice or structured extraction Automated exact match is faster, cheaper, and unbiased. An LLM judge or human review adds cost for no gain.
Safety-critical or regulated domains such as medical, legal, or financial advice Automated metrics and judges cannot be the sole signal. Human expert review is mandatory ground truth.
Very early prototyping with no eval set or labels yet A full three-mode harness is overkill. Start with a handful of manual checks before investing in infrastructure.
Extremely high volume where human review of everything is proposed Humans cannot scale to millions of outputs. Reserve them for a stratified sample and calibration only.
Quick scan · 2–4 min
Everything important, quickly.
3 min: name the three modes, give one strength and one weakness for each, and land the complementary-coverage point with a concrete example per mode.
Real-world examples
Real products, models, and research that use this idea.
1RAGAS combines automated faithfulness scoring with LLM-as-judge calls, while teams still hold out a human-labelled set for calibration.
2OpenAI Evals and Promptfoo run deterministic checks plus judge-based scoring in continuous integration before any model ships.
3Chatbot Arena pairs automated Elo aggregation with large-scale human preference votes as the ground-truth signal.
4LangSmith lets teams layer rule-based evaluators, an LLM judge like GPT-5.5 or Claude Opus 4.7, and human review queues in one workflow.
5Anthropic and similar labs gate frontier releases on automated benchmarks plus extensive human red-teaming, not on judges alone.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you decide which eval mode to apply to a given quality dimension?
Sign in or go Pro to see the approach for this follow-up.
QHow do human labels and an LLM judge relate in a mature eval loop?
Sign in or go Pro to see the approach for this follow-up.
QWhy does drawing your eval set only from production traffic cause problems?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating the three eval modes as interchangeable. They cover different failure classes, so picking only one leaves a blind spot the others would have caught.