Modern LLM evaluation rests on **three complementary modes**, each filling a gap the others leave. The question asks which statements correctly describe how they fit together.
Four options are true. Automated metrics (exact match, F1, BLEU, ROUGE, pass@k) are fastest and cheapest, but only cover well-defined, checkable properties like a string match or a passing test. LLM-as-judge scales to nuanced dimensions (tone, helpfulness, faithfulness) that no rule can capture. Human evaluation is the gold standard for calibration but is slow and expensive. And human spot-checks are best used for periodic calibration, not as your main way to scale.
Two options are false. The claim that all three measure the same signal and are interchangeable is wrong: each covers different properties at different reliability, which is exactly why you combine them.
The claim that LLM judges are always unbiased and never need human validation is also wrong. Judges carry documented biases (position, self-preference, length) and must be checked against human labels.
The mental model: automate the checkable, let a judge score the nuanced, and reserve scarce human attention to calibrate the judge.
The three eval modes are **complementary**, not redundant. Each covers a different slice of quality at a different cost and reliability.
Four statements are correct. Automated metrics are fastest and cheapest, but only score well-defined, checkable properties. LLM-as-judge scales to nuanced dimensions that rules cannot reach. Human evaluation is the slow, expensive gold standard. Human spot-checks belong in periodic calibration, not as the primary scaling path.
Two statements are wrong. The modes do not measure one shared signal, and they are not interchangeable. Exact match cannot judge tone. A judge cannot verify a unit test cheaper than running it. Humans cannot score a million outputs nightly.
The second false claim is that judges are always unbiased and never need validation. Judges carry systematic error: position bias, self-preference, length bias. You must measure judge-human agreement and recalibrate.
The production pattern stacks all three. Automate everything checkable first. Send the rest to an LLM judge. Hold out a small human-labelled set to calibrate the judge with Cohen's kappa, and recalibrate as model versions drift.
Quick reference
TL;DR
The three eval modes are complementary, not interchangeable: automated is fast but narrow, judges scale nuanced scoring with bias, humans are the slow gold standard for calibration.
Think of grading a huge pile of student essays three ways. A spell-checker is instant and free but only catches typos and word counts, never whether the argument is good (automated metrics). A smart teaching assistant can read for tone and reasoning at scale, but has quirky preferences and sometimes marks confident nonsense as good (LLM-as-judge). The professor reads carefully and is the real authority, but can only grade a handful, so you use them to spot-check and calibrate the assistant. Nobody pretends the spell-checker, the assistant, and the professor all measure the same thing or give interchangeable grades. And nobody assumes the assistant is always fair without checking its marks against the professor's.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tasks with cheap, exact ground truth such as classification or structured extraction Automated exact-match scoring is faster, cheaper, and unbiased. Adding a judge or humans wastes effort.
Safety-critical or regulated outputs such as medical, legal, or financial advice Judge bias and weak factual grounding make automation unsafe alone. Human expert review is mandatory.
Very early prototyping with no labelled data and no eval budget yet A full three-mode stack is overkill. Start with a handful of human spot-checks before building a harness.
Open-ended subjective quality like tone, creativity, or empathy Automated metrics like BLEU cannot capture these. Lean on judge plus human calibration instead.
Quick scan · 2–4 min
Everything important, quickly.
4 min: walk the three modes (automated, judge, human), each one's ceiling, why they are complementary not interchangeable, why judges need validation, and the production stack that composes all three.
Real-world examples
Real products, models, and research that use this idea.
1RAGAS combines deterministic context-recall metrics with LLM-judge faithfulness scoring, validated against human labels on a held-out set.
2OpenAI Evals lets teams mix exact-match graders with model-graded (judge) checks in one harness.
3LangSmith ships rule-based evaluators plus LLM-judge evaluators and surfaces judge-human agreement as a first-class metric.
4Chatbot Arena uses large-scale human pairwise votes as the calibration ground truth that automated leaderboards are checked against.
5Promptfoo runs assertion-based automated checks alongside judge graders so cheap deterministic tests gate before the expensive judge runs.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide how to route a given eval property to automated, judge, or human?
Sign in or go Pro to see the approach for this follow-up.
QHow do you measure whether your LLM judge actually agrees with human labels?
Sign in or go Pro to see the approach for this follow-up.
QIf you can only afford to expand one of the three modes, which and why?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating the three modes as interchangeable, or trusting an LLM judge as bias-free ground truth. They cover different properties at different cost and reliability, and judges always need human calibration.