Design a cost efficient LLM-as-judge eval strategy for a system processing 1M queries/day
Your LLM product processes 1 million queries per day. You want to maintain continuous quality monitoring using LLM-as-judge, but evaluating every query with GPT-4 would cost $10,000/day. Design a cost efficient eval strategy that maintains meaningful quality signal.
Sample a stratified slice instead of the full set, route slices to tiered judges, cache deterministic scores, and pay for human eval only on a small high-stakes holdout. Layering these cuts cost by roughly 99 percent.
Imagine a factory making a million chocolates a day. You cannot taste every one. So you grab a fair handful from each flavor line (stratified sampling) instead of only the first tray. A cheap helper sniffs each chocolate first, and only the suspicious ones go to the expensive expert taster (tiered judging). If two chocolates are identical, you reuse the earlier verdict (caching). And every single day you re-taste the same fixed reference box, so you can tell whether quality is truly drifting or whether you just sampled a weird batch. The pricey human expert is reserved for the few chocolates that could hurt someone if they are wrong.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
At 1 million queries per day, judging every output with a frontier model is not just expensive, it is the wrong mental model. You are not trying to assign a grade to every query. You are trying to buy a trustworthy, low-variance estimate of system quality, broken down by the slices that drive decisions, for the smallest spend that keeps that estimate reliable.
That reframing turns the design into a budget-allocation problem. Every lever below trades a little statistical power for a lot of cost. The art is allocating the saved budget toward the slices and failure modes that actually matter, and keeping a stable anchor so you can tell a real regression apart from a shift in what your users happened to ask today.
The full design has four cost levers stacked on a stable baseline: stratified sampling to shrink volume, a tiered cheap-first cascade to shrink per-item cost, deterministic caching to eliminate repeats, and a frozen canary to anchor the trend. A small recurring human audit sits underneath all of it as the calibration ground truth. The sections below take each in turn, then a worked example puts numbers on the whole pipeline.
Stratified sampling: the dominant cost lever
The single biggest saving comes from not judging the full set. Sampling 0.5 to 1 percent of traffic, roughly 5,000 to 10,000 queries, drops the bill by about 99 percent on its own. Everything else is a refinement on top of this.
The key word is stratified, not uniform. Production traffic is heavy-tailed: a handful of query types dominate volume while the high-value slices (a regulated workflow, a paying-tier segment, a new feature) are tiny. Uniform random sampling spends almost all its draws on the boring head and gives the important tail too few samples to detect a regression. The mean stays green while a 0.5 percent slice quietly rots.
Stratify by query type, user segment, language, and a cheap response-risk score. Then allocate samples per stratum by Neyman allocation, where the count scales with stratum size times within-stratum variance, and upweight by business value. A slice that is 0.5 percent of traffic can rightly claim 20 percent of the eval budget.
The sizing question is the one to get right. Pick the smallest quality regression you actually need to catch per slice, the minimum detectable effect, and size each stratum so its confidence interval is narrower than that. A coarse rule of thumb is at least 30 samples per stratum per day to compute a stable mean, but the principled answer is to solve for the interval width you need on the metric that triggers a rollback. Slices with high score variance or high business stakes get more draws; quiet, low-variance slices get fewer. Sampling is not a fixed percentage, it is a per-slice budget driven by how confident you need to be about that slice.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- RAGAS and DeepEval support cheap-judge tiers and per-slice metric breakdowns for sampling-based production eval rather than full-set sweeps.
- LangSmith and Braintrust run sampled online eval with frozen reference datasets as a daily canary alongside live traffic scoring.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide how many samples to draw per stratum?
Use Neyman allocation: samples per stratum scale with stratum size times within-stratum standard deviation, then upweight by business value. Solve for the budget that hits a target confidence interval width on the decision metric, not a fixed per-slice count.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Evaluating a uniform random sample instead of a stratified one. The mean looks fine while a small high-value slice silently regresses, because rare segments get almost no draws.
60 second bullets to scan on the way to the call.
Stratified versus uniform sampling and why the mean hides slice regressions
Neyman allocation: weighting samples by slice value and variance, not volume
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.