Why the eval CI bill becomes a real budget line and how to keep it bounded

Flashcard·Medium·4.0 · 0·~30s·Asked atAirbnb Copy Ai Sigmoid

Attempt it

Three levers.

Tier the set: a small (20-50 example) smoke set runs on every push, the full set runs on PRs that touch tagged files (prompts, retrievers, system messages), and the full plus extended runs nightly on main. Most pushes do not need the full set.

Cheap judge, calibrated: use a smaller cheaper model for the judge (Haiku, Mini, Flash) and periodically calibrate its agreement with a frontier judge or with humans on a 50-example calibration set. The cost difference between Sonnet as judge and Haiku as judge is often 5-10x for a small bias trade-off.

Reuse with caching: when only the prompt changed, the baseline generations are still valid; only re-run the candidate. Cache baseline outputs keyed on prompt-hash + input-hash + model-id.

The outcome should be eval-CI under 5 percent of production LLM spend; past that, the tier or cache discipline slipped.

Your answer

Evaluation in CI is a powerful guard rail and also a hidden cost driver. Every PR that runs a 200-example golden set with an LLM judge pays for one generation per example plus one judge call per example. At even a few dollars per full run, multiplied by every PR and every push, the monthly eval bill can rival production. The first lever is tiering the eval set by what triggered it. Most pushes change a small piece of unrelated code and do not need a full eval. A small smoke set (twenty to fifty examples) runs on every push as a sanity check. The full golden set only runs on PRs that touch the prompt files, the retriever code, or the system messages. The extended set including edge cases and slow tests runs nightly against main. The second lever is swapping the judge model. Frontier judges like Claude Sonnet are accurate but expensive. A cheaper judge like Claude Haiku, GPT-5 Mini, or Gemini Flash is usually within a few points of agreement on the same rubric. Calibrate the cheap judge against the frontier judge on a 50-example calibration set every couple of weeks to confirm the agreement holds. The third lever is caching. When only the prompt changed, the baseline model already generated those outputs on the last run. Cache them keyed on prompt-hash, input-hash, and model-id, and only regenerate the candidate side. With all three levers, eval-CI usually stays under 5% of production LLM spend.

**Three levers** bound the eval-CI bill without sacrificing the signal that justified running evals in CI to begin with. **Tier the set by trigger.** A 20-50 example smoke set runs on every push as a fast sanity check. The full 200-example golden set runs only on PRs that touch the tagged surfaces (prompt files, retriever configs, system messages, model-router code). The full plus extended set runs nightly on main. Most pushes never touch the tagged surfaces, so most pushes only pay for the smoke set. **Calibrated cheap judge.** Swap the frontier judge for a cheaper model (Claude Haiku, GPT-5 Mini, Gemini Flash) and pay roughly 5-10x less per judge call. Maintain a small calibration set (around 50 examples) where both the cheap judge and a frontier judge or a human have rated the same outputs. Compute Cohen's kappa or simple agreement weekly; if the cheap judge drifts below the agreement threshold, retrain the rubric prompt or upgrade the judge. **Cache baseline generations.** When a PR changes only the prompt, the baseline model's outputs from the previous main run are still valid for the same inputs. Cache them keyed on `(prompt_hash, input_hash, model_id, model_version)` and skip regeneration. Only the candidate side runs against the changed prompt. Cache invalidation is automatic when any keyed dimension changes. **Target.** Eval-CI spend below 5% of production LLM spend. Above that, one of the three levers slipped and is worth a short investigation.

TL;DR

Bound eval-CI cost by tiering the set by trigger, swapping the frontier judge for a calibrated cheap judge, and caching baseline generations so only the candidate is regenerated per run.

Memory aid

Easy to grasp

Imagine a bakery that tastes every batch of bread before shipping. Tasting every loaf with the head pastry chef on every batch is expensive and slow. So the bakery sets up tiers. A quick sniff test by a junior baker checks every single batch. The full taste panel by the senior team only happens when the recipe or flour supplier changed. The exhaustive blind tasting with outside judges runs once a night against the master recipe. Most days the quick test catches the issue, the senior team only gets pulled in when something real changed, and the head chef saves their time for genuinely new recipes. The bakery still ships safe bread; it just stops burning the senior team's salary on toast.

Key concepts

LLMOps

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Eval-CI on LLM applications is one of the most valuable engineering investments a team can make. It catches prompt regressions before they ship, gives a defensible answer to 'is the new model actually better', and forces honest measurement of changes that would otherwise be argued about in pull-request comments. It is also one of the easiest investments to under-budget, because the per-run cost looks small until multiplied by every push from every engineer.

A 200-example golden set with an LLM-as-judge rubric typically costs a few dollars per run. Run that on every PR, with every push to that PR, against a baseline plus a candidate, plus a frontier judge that costs more than the model being evaluated, and the monthly bill can rival a small production workload by month six. The temptation when finance notices is to scale eval back, which is exactly the wrong response because it removes the guard rail that prevents production regressions.

The right response is to keep the coverage and cut the cost per run. Three levers do most of the work: tiering the eval set by what changed, swapping the judge for a calibrated cheap model, and caching baseline generations across runs.

Tiering by trigger: most pushes do not need the full set

The naive setup runs the full eval on every push to every PR. That guarantees coverage and burns budget on coverage nobody needs. The vast majority of pushes touch test files, docs, build configs, or unrelated application code that has zero chance of affecting LLM output.

The tiering pattern uses changed-path filters as the trigger. A smoke set of 20-50 examples runs on every push as a fast sanity check. This catches the obvious 'I broke the import' or 'the prompt template now has a syntax error' regressions in under a minute. The full 200-example golden set runs only on PRs that touch tagged surfaces: prompt files, retriever configs, system messages, model-router code, tool definitions. A PR touching multiple tagged surfaces runs the full set plus a small regression panel for cross-surface interactions.

The extended set, including slow examples, rare cohorts, and adversarial probes that push PR latency too high, runs nightly against main as a backstop. If the nightly catches a regression that the per-PR runs missed, that is signal to add a new tagged-surface filter or to enlarge the per-PR golden set.

In practice, the tiering shift alone usually drops eval-CI spend by 60-80% because most PRs are infrastructure or unrelated application changes.

Cheap judge with calibration is a discipline, not a downgrade

Caching: the highest payoff, the easiest to mess up

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Promptfoo and Inspect AI both support partial-set runs and result caching out of the box, which lets a team implement the tiering and caching pattern without writing custom orchestration.
A team using Claude Sonnet 4 in production runs Claude Haiku as the eval judge with a weekly 50-example calibration against Sonnet; agreement stays above 0.78 kappa and the judge bill drops by roughly 8x.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you handle the case where the smoke set passes but the full set on the next PR catches a regression introduced earlier?

Run the full set nightly on main as a backstop; tag the offending commit when the nightly fails and bisect with the cached baselines to localize the regression cheaply.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Running the full golden set with the frontier judge on every push. The cost compounds into a production-sized line item without delivering more signal than a tiered approach.

60 second bullets to scan on the way to the call.

Tier by trigger: smoke on every push, full on tagged paths, extended nightly
Cheap judge plus a calibration set with periodic kappa check

Primary sources. Browse if you want the original framing.