Design a cost-efficient LLM-as-judge eval strategy for a system processing 1M queries/day

At 1 million queries a day, judging every output with a frontier model is too expensive. The trick is that you do not need to judge everything to know whether quality is holding. You need a representative signal, not a full census. The biggest lever is **stratified sampling**. Instead of evaluating all 1M queries, you evaluate a small slice, maybe 5,000 to 10,000 per day. Stratified means you draw a fair share from each important group (query type, user segment, language) so rare but important slices still get covered. The second lever is tiered judges. Use a cheap judge for routine traffic and reserve the expensive frontier judge for high-stakes slices like safety or billing. Most traffic is boring, so most of it can go to the cheap tier. The third lever is caching. At temperature 0 the same input gives the same score, so identical or repeated queries reuse a stored result instead of paying again. Finally, keep a frozen canary set of a few hundred fixed examples scored every day. That gives a stable baseline that does not wobble when your traffic mix shifts. Reserve human eval for the small, high-stakes holdout where a wrong judge call is costly.

The strategy layers four cost levers on top of a stable baseline. Each attacks a different part of the bill. **Stratified sampling first.** Do not judge all 1M queries. Sample 0.5 to 1 percent, stratified by query type, user segment, and response characteristics. This alone cuts cost by about 99 percent while preserving slice coverage. Uniform sampling would let rare high-value slices vanish into the mean. **Tiered judging and a cheap-first filter.** Run a cheap model or fine-tuned classifier as a first pass. It flags suspect responses and routes only those to the expensive frontier judge. Route high-stakes slices like safety, regulated, and user-reported traffic to the strong judge. Route routine traffic to a cheap judge like Haiku or GPT-4o-mini. **Cache deterministic scores.** At temperature 0 with a pinned model, identical inputs yield identical scores. Cache by input hash so repeated and near-duplicate queries never get re-evaluated. **Freeze a canary set.** Score the same 500-example holdout daily. The absolute baseline stays stable even as traffic shifts. Pay for human eval only on that holdout and on flagged high-stakes cases. Net effect: roughly $50 to $150 per day instead of $10,000.