Order the three eval approaches from cheapest/fastest to most expensive/slowest
- 1LLM-as-judge scoring on a sampled subset
- 2Periodic human spot check calibration triggered by LLM judge score drift
- 3Human evaluation with domain experts
- 4Automated metrics (format checks, regex, string matching)
Layer eval cheapest first: automated checks on everything, LLM-judge on a sample, event-triggered human spot-checks, then expensive expert review as the final ground truth.
Imagine grading a huge pile of essays. First you run a spell-checker on every single one because it is instant and free. Then a smart teaching assistant skims a random handful to rate quality, which costs a little. If the assistant's ratings suddenly look weird, you pull a few essays and have a senior teacher double-check that the assistant is still grading sensibly. Finally, for the trickiest essays, you bring in a subject expert who reads slowly and charges by the hour. You always start with the cheap, fast checks on everything and save the slow, expensive humans for the small slice that truly needs them. That ordering keeps the bill small while still catching the problems that matter.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Layered evaluation is the dominant production pattern in 2026 because no single eval method is simultaneously cheap, fast, and trustworthy. Automated metrics are instant and free but blind to quality. Human experts are the gold standard but slow and expensive. LLM-as-judge sits in between. The engineering move is to arrange these methods as a funnel ordered by cost, run the cheapest tier on everything, and let each pricier tier see only the shrinking slice the cheaper one could not resolve.
This question asks you to order four eval approaches from cheapest and fastest to most expensive and slowest. The correct sequence is automated metrics, then LLM-as-judge on a sample, then an event-triggered human spot-check, then full human expert evaluation. The deep dive walks each tier, why it lands where it does, and the failure modes of getting the order wrong.
Tier one: automated metrics on everything
Automated metrics are the cheapest tier by a wide margin. Format validators, regex matchers, JSON-schema checks, exact-match comparisons, and length bounds run in well under a millisecond and cost effectively nothing per call. There is no API round-trip, no token bill, and no human in the loop. The check is pure local computation, so it scales linearly with traffic and never becomes a budget line item the way a paid judge or a human reviewer does.
Because they are free and instant, you run them on every output, in CI before merge and on the live stream in production. Their job is to catch the deterministic failures: malformed JSON, a refused response, output in the wrong language, a missing required field, an answer that blew past the length cap. These checks are also the natural place for hard policy gates, a banned-phrase list, a PII regex, a safety keyword filter, because a categorical rule is exactly what you want when the failure is binary rather than a matter of degree.
Their weakness is that they only check structure, never quality. A perfectly formatted, schema-valid answer can still be unhelpful, off-topic, or factually wrong, and no regex will notice. The false-negative rate on quality is high; the false-positive rate on structure is near zero. That asymmetry is exactly what makes them a good first gate. That is precisely why they go first: they cheaply remove the obvious failures so that no expensive downstream tier wastes a paid call on output that was already broken on syntax. Think of this tier as triage that costs nothing to run and removes the cases nobody needs a model or a human to adjudicate.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangSmith and Braintrust pipelines run deterministic assertions on every trace, then schedule LLM-judge scoring on a sampled slice rather than the full log.
- RAGAS layers cheap context-precision checks ahead of expensive judge-based faithfulness scoring on RAG outputs.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you decide the sampling rate for the LLM-judge tier?
Balance variance against cost. Estimate the score standard deviation, pick a sample size that bounds the confidence interval, and stratify by query type so rare segments keep enough examples to be meaningful.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Running expensive human review or LLM-judge over the whole dataset when cheap automated checks could have filtered most of it first. Order by cost ascending, narrow the candidate set at each tier.
60 second bullets to scan on the way to the call.
Why eval is layered rather than a single gate
Cost and latency profile of automated metrics
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.