Which strategies reduce LLM-as-judge evaluation cost at scale without sacrificing reliability?
Cut judge cost by evaluating fewer items (stratified sampling, caching) or paying less per item (tiered judges, hierarchical cheap filter first). Max-capability-everywhere and temperature tweaks do neither.
Imagine grading a million student essays with one expensive expert. You cannot afford to read them all, so you get smart about it. First, you read a fair sample from each class instead of every paper (sampling). Second, you keep a folder of essays you have already graded so identical ones are never re-read (caching). Third, you let a cheap teaching assistant skim everything and only pass the borderline or important ones to the expensive expert (tiered and hierarchical eval). What does NOT save money: making the expensive expert read every single paper, or telling them to grade a little more randomly. Neither reads fewer essays nor makes each grade cheaper, so the bill stays the same or grows.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LLM-as-judge is the default way to measure LLM product quality at scale in 2026, but at production volume the economics bite. A system serving millions of requests, evaluated with a strong judge model like a Claude Opus or GPT-5 class model, can run a judge bill that rivals the production inference bill itself. So a staff-level eval engineer treats judge cost as a first-class design constraint, not an afterthought.
The insight that organizes every valid strategy is that judge spend equals call volume multiplied by cost per call. You can only push down one of those two factors: reduce the number of evaluations, or reduce the price of each one. Reliability is the hard constraint, because a cheaper eval that no longer agrees with human judgment is not a saving, it is a broken instrument.
The trap candidates fall into is treating cost and reliability as opposed, then either over-spending to feel safe or cutting corners that silently bias the numbers. The senior move is to recognize that the two goals are compatible when you respect a single rule: any cost cut is only valid if the cheaper path is validated to still agree with the expensive ground truth on the slices that matter. This deep dive walks the two levers, the four strategies that pull them, the token-level optimizations that stack on top, and the two distractors that pull neither.
The two levers: volume and unit cost
Every defensible cost optimization reduces either the count of judge calls or the price of each call. Naming the lever for each option is the fastest way to separate the real strategies from the distractors during an interview.
Volume-reducing strategies include stratified sampling (score a subset, not the full population) and caching (never score the same input twice). Unit-cost-reducing strategies include tiered judging (route easy slices to a cheaper model). Hierarchical eval pulls both at once: a cheap filter cuts volume to the expensive stage, and the filter itself runs at low unit cost.
The two distractors fail this test immediately. Routing every query to the most capable judge increases volume at the highest possible unit cost, which is the exact baseline these strategies fight. Tuning temperature to 0.5 changes neither factor: the judge still runs on every item, and a single judge call costs the same in tokens regardless of its temperature. So it cannot reduce spend, and the variance it adds actively hurts reliability.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Strategy | Lever | Main reliability risk |
|---|---|---|
| Stratified sampling | Fewer evals | Under-sampling high-variance slices inflates noise |
| Caching | Fewer evals | Stale scores if judge or rubric version changes |
| Tiered judges | Cheaper per eval | Cheap judge disagrees with humans on routed slices |
| Hierarchical filter | Both | Cheap filter drops true positives before escalation |
| Max judge everywhere | None (cost ceiling) | Spend scales linearly with traffic |
Real products, models, and research that use this idea.
- Braintrust and LangSmith let teams sample a percentage of production traces for offline judge eval rather than scoring every request.
- Promptfoo and DeepEval support caching judge results keyed on input plus config so reruns skip already-scored items.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide how many examples to sample per stratum?
Use Neyman allocation: assign more sample budget to slices with higher variance or higher business stakes. Compute per-slice confidence intervals and increase a slice's sample size until its interval is tight enough to detect the regression you care about.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reaching for the strongest judge on every slice to be safe. That maximizes spend without improving the slices where a cheap judge already agrees with humans, and the bill scales linearly with traffic.
60 second bullets to scan on the way to the call.
The two cost levers: fewer evaluations versus cheaper per evaluation
Stratified sampling and how to allocate sample budget across slices
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.