Your LLM product processes 1 million queries per day. You want to maintain continuous quality monitoring using LLM-as-judge, but evaluating every query with GPT-4 would cost $10,000/day. Design a cost-efficient eval strategy that maintains meaningful quality signal.
Free · 2 AI evals / day
At 1 million queries a day, judging every output with a frontier model is too expensive. The trick is that you do not need to judge everything to know whether quality is holding. You need a representative signal, not a full census.
The biggest lever is **stratified sampling**. Instead of evaluating all 1M queries, you evaluate a small slice, maybe 5,000 to 10,000 per day. Stratified means you draw a fair share from each important group (query type, user segment, language) so rare but important slices still get covered.
The second lever is tiered judges. Use a cheap judge for routine traffic and reserve the expensive frontier judge for high-stakes slices like safety or billing. Most traffic is boring, so most of it can go to the cheap tier.
The third lever is caching. At temperature 0 the same input gives the same score, so identical or repeated queries reuse a stored result instead of paying again.
Finally, keep a frozen canary set of a few hundred fixed examples scored every day. That gives a stable baseline that does not wobble when your traffic mix shifts. Reserve human eval for the small, high-stakes holdout where a wrong judge call is costly.
The strategy layers four cost levers on top of a stable baseline. Each attacks a different part of the bill.
**Stratified sampling first.** Do not judge all 1M queries. Sample 0.5 to 1 percent, stratified by query type, user segment, and response characteristics. This alone cuts cost by about 99 percent while preserving slice coverage. Uniform sampling would let rare high-value slices vanish into the mean.
**Tiered judging and a cheap-first filter.** Run a cheap model or fine-tuned classifier as a first pass. It flags suspect responses and routes only those to the expensive frontier judge. Route high-stakes slices like safety, regulated, and user-reported traffic to the strong judge. Route routine traffic to a cheap judge like Haiku or GPT-4o-mini.
**Cache deterministic scores.** At temperature 0 with a pinned model, identical inputs yield identical scores. Cache by input hash so repeated and near-duplicate queries never get re-evaluated.
**Freeze a canary set.** Score the same 500-example holdout daily. The absolute baseline stays stable even as traffic shifts. Pay for human eval only on that holdout and on flagged high-stakes cases. Net effect: roughly $50 to $150 per day instead of $10,000.
Quick reference
TL;DR
Sample a stratified slice instead of the full set, route slices to tiered judges, cache deterministic scores, and pay for human eval only on a small high-stakes holdout. Layering these cuts cost by roughly 99 percent.
Imagine a factory making a million chocolates a day. You cannot taste every one. So you grab a fair handful from each flavor line (stratified sampling) instead of only the first tray. A cheap helper sniffs each chocolate first, and only the suspicious ones go to the expensive expert taster (tiered judging). If two chocolates are identical, you reuse the earlier verdict (caching). And every single day you re-taste the same fixed reference box, so you can tell whether quality is truly drifting or whether you just sampled a weird batch. The pricey human expert is reserved for the few chocolates that could hurt someone if they are wrong.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Safety-critical or regulated slices where a missed failure is costly Sampling can skip the rare bad case. These slices need near-full coverage and human adjudication.
Very low traffic products with only hundreds of queries per day Sampling adds variance for no real savings. Just judge the full set with one good model.
A brand new product with no historical baseline or judge calibration yet You cannot stratify or trust a cheap judge before you know the slices and have human labels.
One-off offline benchmarking on a small fixed eval set The set is already small and stable. Tiering and sampling add complexity with no payoff.
Quick scan · 2–4 min
Everything important, quickly.
6 min: reframe as a measurement-budget problem, then walk the four levers (stratified sampling, tiered cascade, caching, canary) plus where human eval earns its cost and the rough cost math from $10k/day down to about $100/day.
Real-world examples
Real products, models, and research that use this idea.
1RAGAS and DeepEval support cheap-judge tiers and per-slice metric breakdowns for sampling-based production eval rather than full-set sweeps.
2LangSmith and Braintrust run sampled online eval with frozen reference datasets as a daily canary alongside live traffic scoring.
3Promptfoo caches judge results by input hash at temperature 0, so re-running an unchanged eval suite costs near zero.
4Langfuse and Arize sample a fraction of production traces for LLM-judge scoring and reserve human review for flagged outliers.
5Patronus and Galileo offer cheap fine-tuned guardrail judges as a first-pass filter that escalates only suspect outputs to a frontier judge.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide how many samples to draw per stratum?
Sign in or go Pro to see the approach for this follow-up.
QHow do you keep the cheap first-pass judge from missing real failures?
Sign in or go Pro to see the approach for this follow-up.
QWhen is a cached score stale even though the input is identical?
Sign in or go Pro to see the approach for this follow-up.
QHow would you detect that your sampling itself has become biased?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Evaluating a uniform random sample instead of a stratified one. The mean looks fine while a small high-value slice silently regresses, because rare segments get almost no draws.