Which properties correctly describe the complementary roles of the three LLM evaluation modes?
The three eval modes are complementary, not interchangeable: automated is fast but narrow, judges scale nuanced scoring with bias, humans are the slow gold standard for calibration.
Think of grading a huge pile of student essays three ways. A spell-checker is instant and free but only catches typos and word counts, never whether the argument is good (automated metrics). A smart teaching assistant can read for tone and reasoning at scale, but has quirky preferences and sometimes marks confident nonsense as good (LLM-as-judge). The professor reads carefully and is the real authority, but can only grade a handful, so you use them to spot-check and calibrate the assistant. Nobody pretends the spell-checker, the assistant, and the professor all measure the same thing or give interchangeable grades. And nobody assumes the assistant is always fair without checking its marks against the professor's.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LLM evaluation is hard because there is rarely a single correct output string to match against. Classic machine learning eval leans on a gold label and an exact comparison, but a generated answer can be correct in a thousand surface forms and wrong in a thousand subtle ways. Quality is multi-dimensional: faithfulness, relevance, completeness, tone, safety. No one measurement tool covers all of it well, so modern eval composes three complementary modes, each strong where the others are weak.
The three modes form a deliberate division of labor. Automated metrics own the cheap, checkable end. LLM-as-judge owns the nuanced middle that needs general intelligence but tolerates some noise. Human evaluation owns the high-stakes, ambiguous end and serves as the calibration anchor for everything else. Each mode trades cost, coverage, and reliability differently, and the art of eval engineering is composing them so the strengths of one cover the blind spots of another.
This question lists six statements about how those modes fit together. Four describe the real division of labor correctly, and two are misconceptions that show up constantly in interviews and in shipped systems. The deep dive walks each mode, its ceiling, why the two false claims fail, and the production pattern that stacks all three into one harness.
Automated metrics: fast, cheap, narrow
Automated metrics are deterministic functions of the output and a reference. Exact match and F1 for classification and extraction, pass@k for code, BLEU and ROUGE and METEOR and BERTScore for generation, perplexity as an intrinsic proxy. They run in milliseconds, cost almost nothing, and are perfectly reproducible. Because they are deterministic, they make excellent regression gates in CI: a score change between two model versions is a real signal, not sampling noise.
Their ceiling is coverage. They only score properties with a checkable definition. A unit test can confirm code runs, but no n-gram overlap metric can tell you whether an answer is faithful to retrieved context or whether the tone fits a support-chat persona. Surface-overlap metrics are especially brittle on open-ended generation, where a paraphrase that is perfectly correct can score low simply because it shares few n-grams with the reference. That brittleness is the whole reason the field moved beyond BLEU and ROUGE for anything subjective.
This is exactly what option A states, and it is correct. The senior discipline is to push everything checkable into this tier first, because it is the cheapest and most reliable, and only escalate what genuinely needs intelligence to the more expensive modes. A well-designed harness routes the largest possible share of failures to deterministic graders so the judge and human tiers only ever see what truly needs them.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- RAGAS combines deterministic context-recall metrics with LLM-judge faithfulness scoring, validated against human labels on a held-out set.
- OpenAI Evals lets teams mix exact-match graders with model-graded (judge) checks in one harness.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide how to route a given eval property to automated, judge, or human?
Ask if the property has a checkable definition (automate it), needs general intelligence but tolerates noise (judge), or is high-stakes and ambiguous (human). Route by checkability, stakes, and tolerance for noise, then use humans to calibrate the judge tier.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating the three modes as interchangeable, or trusting an LLM judge as bias-free ground truth. They cover different properties at different cost and reliability, and judges always need human calibration.
60 second bullets to scan on the way to the call.
What automated metrics cover and where their ceiling is
Why LLM as judge fills the nuance gap
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.