Which strategies reduce LLM-as-judge evaluation cost at scale without sacrificing reliability?

Attempt it

Quick reference

TL;DR

Cut judge cost by evaluating fewer items (stratified sampling, caching) or paying less per item (tiered judges, hierarchical cheap filter first). Max-capability-everywhere and temperature tweaks do neither.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine grading a million student essays with one expensive expert. You cannot afford to read them all, so you get smart about it. First, you read a fair sample from each class instead of every paper (sampling). Second, you keep a folder of essays you have already graded so identical ones are never re-read (caching). Third, you let a cheap teaching assistant skim everything and only pass the borderline or important ones to the expensive expert (tiered and hierarchical eval). What does NOT save money: making the expensive expert read every single paper, or telling them to grade a little more randomly. Neither reads fewer essays nor makes each grade cheaper, so the bill stays the same or grows.

Key concepts

LLM evaluation LLM system design Inference optimization

Quick scan · 2–4 min

Everything important, quickly.

5 min: the two cost levers, the four valid strategies mapped to each lever, the reliability risk that gates every one, and why the two distractors save nothing.

Side-by-side

Strategy	Lever	Main reliability risk
Stratified sampling	Fewer evals	Under-sampling high-variance slices inflates noise
Caching	Fewer evals	Stale scores if judge or rubric version changes
Tiered judges	Cheaper per eval	Cheap judge disagrees with humans on routed slices
Hierarchical filter	Both	Cheap filter drops true positives before escalation
Max judge everywhere	None (cost ceiling)	Spend scales linearly with traffic

Real-world examples

Real products, models, and research that use this idea.

Braintrust and LangSmith let teams sample a percentage of production traces for offline judge eval rather than scoring every request.
Promptfoo and DeepEval support caching judge results keyed on input plus config so reruns skip already-scored items.
Anthropic and OpenAI batch APIs offer roughly 50 percent discounts for asynchronous judge eval jobs run overnight.
Patronus and Galileo ship cheap small-model guardrail filters that escalate only flagged outputs to a stronger judge.
RAGAS uses claim-level entailment with a configurable judge, letting teams swap a cheaper model on low-stakes faithfulness slices.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you decide how many examples to sample per stratum?

Sign in or go Pro to see the approach for this follow-up.

QWhat must hold for a hierarchical cheap filter first design to preserve reliability?

Sign in or go Pro to see the approach for this follow-up.

QWhen does caching judge scores produce silently wrong numbers?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.