Running LLM-as-judge on your full eval set costs $200 per run. The team runs evals 10 times a day during development. How do you cut cost without losing signal?
The problem is $2,000 per day: 10 eval runs at $200 each. Most of that spend is unnecessary during development, where the goal is fast feedback, not final judgment.
The fix has three parts. First, stratified sampling: instead of running the full eval set, run a 20% subset that covers every query type proportionally. This drops each run from $200 to $40 while preserving signal across categories. The full set is reserved for pre-merge gates.
Second, caching. If the same input and the same model produce the same output at temperature 0, the judge score is deterministic. Cache it. When a developer re-runs the eval after changing one prompt, only the affected examples need re-scoring.
Third, hierarchical eval. Fast automated checks (format validation, length bounds, keyword presence) filter out obvious failures before the expensive judge sees them. The judge only scores examples that pass the cheap checks, reducing the sample size further.
The distractors fail for clear reasons. BLEU and ROUGE measure surface word overlap and are nearly meaningless for open-ended generation. Temperature does not reduce cost in token-based pricing. Skipping daily evals removes the developer feedback loop entirely.
**Cost-aware eval** treats the judge budget as a resource allocation problem, not an all or nothing choice.
Stratified sampling is the primary lever. For daily development runs, draw a representative 20% of the eval set, stratified by query type and difficulty. Each run drops from $200 to $40. The stratification ensures every category is represented, so a regression in any one type is detectable even on the smaller sample. Reserve the full eval set for pre-merge and pre-deploy gates where accuracy justifies the cost.
Caching deterministic re-scores is the second lever. At temperature 0, the same input through the same model produces the same output. The judge score for that pair is fixed. Cache it and invalidate only when the model, prompt, or input changes. On a typical development cycle, 60 to 80% of examples are unchanged between runs.
Hierarchical eval is the third lever. Fast automated checks (format validation, length bounds, keyword presence, safety-classifier pass) filter out obvious failures before the judge runs. The judge scores only the examples that pass cheap checks, further reducing the sample.
A fourth option for rapid iteration: use a cheaper, smaller judge model (e.g. a fine-tuned smaller model calibrated against the expensive judge) for development runs and reserve the expensive judge for gates.
BLEU and ROUGE fail for open-ended tasks because surface overlap does not capture semantic quality. Temperature does not reduce cost in per-token pricing. Skipping daily evals breaks the fast-feedback loop that makes iterative development productive.
Quick reference
TL;DR
Stratified sampling for daily runs, caching deterministic re-scores, and reserving the expensive judge for pre-merge gates cuts cost 80% without losing signal across query types.
Imagine you are a food inspector who has to taste 100 dishes at a restaurant every day. Tasting all 100 takes hours and costs a fortune. Instead, you pick 20 dishes that represent every category on the menu (appetizers, mains, desserts). If the 20 taste fine, the restaurant is probably fine. You save the full 100-dish tasting for the annual health inspection. And if a dish has not changed since yesterday, you skip re-tasting it. You get 80 percent of the coverage at 20 percent of the cost.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A pre-launch final evaluation where accuracy on every example matters for the ship decision. Sampling trades accuracy for cost. When the decision stakes are high (launch, major model swap), run the full eval set with the best judge available.
A very small eval set (fewer than 50 examples) where 20% sampling leaves too few examples per category. Stratified sampling requires enough examples per stratum to be statistically meaningful. Below a minimum threshold, run the full set.
A regulatory or compliance evaluation where every example must be evaluated and documented. Sampling is not acceptable when the audit requires coverage of every test case. Accept the cost and run the full eval.
Quick scan · 2–4 min
Everything important, quickly.
Start by framing the $2,000/day cost problem. Walk through stratified sampling (20% with per-category coverage), deterministic caching (hash-keyed, invalidate on change), hierarchical eval (cheap pre-filter then expensive judge), and model tiering (cheap dev judge, expensive gate judge). Quantify the combined savings. Dismantle the distractors: BLEU/ROUGE fail for open-ended tasks, temperature does not reduce cost, skipping evals breaks the feedback loop. Close with the tradeoff: smaller samples detect large regressions but miss subtle ones.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's internal eval infrastructure uses stratified subsets for rapid development iteration and reserves full eval sweeps for model promotion gates.
2OpenAI's eval framework (evals repo) supports caching and incremental evaluation so only changed examples are re-scored between runs.
3Ragas supports configurable sampling rates for RAG evaluation so teams can trade accuracy for cost during development and run full evaluations before releases.
4Production ML teams at Stripe and Uber use hierarchical evaluation pipelines where cheap heuristic checks filter out obvious failures before expensive model-based evaluation runs.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QYour stratified 20% sample shows no regression, but after merging, the full pre-merge eval shows a 5% accuracy drop on one query type. What went wrong?
Sign in or go Pro to see the approach for this follow-up.
QHow do you validate that a cheaper judge model is a good proxy for the expensive one?
Sign in or go Pro to see the approach for this follow-up.
QYour cache hit rate is only 30% because developers change prompts frequently. How do you increase it?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Replacing LLM-as-judge with BLEU/ROUGE to save cost, losing the ability to evaluate open-ended generation quality where surface-overlap metrics are nearly meaningless.