Which strategies reduce LLM-as-judge evaluation cost at scale without sacrificing reliability?

Running an **LLM-as-judge** over a production system can mean millions of judge calls per quarter, and the judge is usually a strong, expensive model. So eval cost becomes a real budget line, and the question is how to cut it without making the scores untrustworthy. There are only two levers. You either evaluate fewer items, or you pay less for each item. Every valid strategy maps to one of those. Four of the options are real wins. Stratified sampling means you score a representative subset of each query type instead of every query, which cuts volume while keeping coverage. Caching stores scores for identical inputs so you never pay twice. Tiered judging sends easy or low-stakes slices to a cheap judge and saves the expensive judge for the hard ones. Hierarchical eval runs a fast cheap filter first and only escalates the items that survive it. The two wrong options do nothing for cost. Running every query through the most capable judge is the most expensive choice, not the cheapest. And changing temperature to 0.5 does not meaningfully reduce token spend, it just adds randomness, which hurts reliability.

**Judge cost reduces along exactly two levers:** fewer evaluations, or cheaper per evaluation. The four correct options each pull one lever. Stratified sampling pulls the volume lever. Score a representative slice of each query type instead of the full set. A 100 to 500 example sample per stratum usually captures the metric with acceptable variance. Caching also pulls volume. If inputs repeat and the judge runs at T=0, store and reuse the score. Identical input never costs twice. Tiered judging pulls the per-item cost lever. Route low-stakes or low-variance slices to a cheap judge. Reserve the expensive judge for high-stakes or high-disagreement slices. Hierarchical eval pulls both. A fast lightweight filter handles the easy majority, and the expensive judge only sees items that survive the filter. The two distractors do neither. Running every query through the most capable judge maximizes cost; it is the baseline these strategies fight. And nudging temperature to 0.5 does not cut the token budget. It just injects sampling noise, which raises variance and hurts reliability. That is backwards from the goal.