Running LLM-as-judge on your full eval set costs $200 per run. The team runs evals 10 times a day during development. How do you cut cost without losing signal?

The problem is $2,000 per day: 10 eval runs at $200 each. Most of that spend is unnecessary during development, where the goal is fast feedback, not final judgment. The fix has three parts. First, stratified sampling: instead of running the full eval set, run a 20% subset that covers every query type proportionally. This drops each run from $200 to $40 while preserving signal across categories. The full set is reserved for pre-merge gates. Second, caching. If the same input and the same model produce the same output at temperature 0, the judge score is deterministic. Cache it. When a developer re-runs the eval after changing one prompt, only the affected examples need re-scoring. Third, hierarchical eval. Fast automated checks (format validation, length bounds, keyword presence) filter out obvious failures before the expensive judge sees them. The judge only scores examples that pass the cheap checks, reducing the sample size further. The distractors fail for clear reasons. BLEU and ROUGE measure surface word overlap and are nearly meaningless for open-ended generation. Temperature does not reduce cost in token-based pricing. Skipping daily evals removes the developer feedback loop entirely.

**Cost-aware eval** treats the judge budget as a resource allocation problem, not an all or nothing choice. Stratified sampling is the primary lever. For daily development runs, draw a representative 20% of the eval set, stratified by query type and difficulty. Each run drops from $200 to $40. The stratification ensures every category is represented, so a regression in any one type is detectable even on the smaller sample. Reserve the full eval set for pre-merge and pre-deploy gates where accuracy justifies the cost. Caching deterministic re-scores is the second lever. At temperature 0, the same input through the same model produces the same output. The judge score for that pair is fixed. Cache it and invalidate only when the model, prompt, or input changes. On a typical development cycle, 60 to 80% of examples are unchanged between runs. Hierarchical eval is the third lever. Fast automated checks (format validation, length bounds, keyword presence, safety-classifier pass) filter out obvious failures before the judge runs. The judge scores only the examples that pass cheap checks, further reducing the sample. A fourth option for rapid iteration: use a cheaper, smaller judge model (e.g. a fine-tuned smaller model calibrated against the expensive judge) for development runs and reserve the expensive judge for gates. BLEU and ROUGE fail for open-ended tasks because surface overlap does not capture semantic quality. Temperature does not reduce cost in per-token pricing. Skipping daily evals breaks the fast-feedback loop that makes iterative development productive.