Build an eval framework from scratch for three LLM products (support bot, code assistant, summarizer). Cover architecture, shared infra, per product customization, and six month failure modes.
Build an eval framework from scratch for three LLM products (support bot, code assistant, summarizer). Cover architecture, shared infra, per product customization, and six month failure modes.
Shared eval runner with per-product configs, versioned storage, results DB, CI gates. Four failure modes: rubric drift, judge deprecation, golden set staleness, cross-product comparison trap.
Imagine three restaurants in the same building sharing one kitchen inspection system. Each restaurant serves different food, so the inspection checklist is different for each: the sushi place checks fish freshness, the bakery checks oven temperatures, the salad bar checks produce sourcing. The shared part is the inspection schedule, the filing system for reports, and the alarm that rings when a score drops. But if the sushi menu changes and nobody updates the checklist, inspections pass while customers get sick. If the inspector retires and nobody trains a replacement, all three restaurants lose their reviewer on the same day. If the filing system only stores last month's scores, nobody notices a slow decline. And if the building manager compares the sushi score to the bakery score and concludes the bakery is worse, that comparison is meaningless because the checklists measure different things.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Building an evaluation framework for multiple LLM products is a systems design problem, not a scripting exercise. The framework must serve three products with fundamentally different quality definitions while sharing enough infrastructure to avoid maintaining three independent eval systems.
The architecture has two layers: a shared execution layer that is product-agnostic, and a per-product configuration layer where each product defines what quality means for its use case. The shared layer handles storage, execution, results, and CI integration. The per-product layer handles golden sets, metric stacks, rubrics, and judge prompts.
The design challenge is not building either layer in isolation. It is anticipating the failure modes that emerge when the framework runs in production for six months: rubric drift, judge model deprecation, golden set staleness, and the cross-product comparison trap. Each of these failures is predictable and preventable with architectural choices made on day one.
Shared infrastructure: the product-agnostic execution layer
The shared layer has four components that every product uses.
Versioned test case storage. Golden sets, rubrics, and judge prompts live in version-controlled storage (typically git). Every eval run records the exact version of the golden set it ran against. This makes runs reproducible: six months later, you can re-run the exact eval that produced a historical score. Without versioning, golden set edits silently invalidate all historical comparisons.
Eval execution engine. The engine reads a product config, loads the golden set, dispatches eval calls (either deterministic metric computation or LLM-as-judge calls), caches results for unchanged test cases, and parallelizes across workers. Caching matters because the code assistant's pass@k evaluations involve executing generated code, which is slow. Parallelism matters because the support bot's multi-turn evals involve sequential conversation turns that cannot be batched naively.
Results database with run comparison. Every eval run writes structured results: per-example scores, aggregate metrics, config versions, timestamps. The database supports run over run comparison (did this PR improve or regress quality?), trend dashboards (is quality drifting over weeks?), and automated alerting (score dropped more than two standard deviations from the 30-day rolling mean). Without structured results, teams resort to spreadsheets and lose the ability to detect slow drift.
CI/CD pre-merge gates. The eval suite runs on every pull request. If the regression gate fails (aggregate score drops below threshold, or any critical test case flips from pass to fail), the merge is blocked. This turns the eval framework from a reporting tool into a quality gate. Teams that skip CI integration discover regressions after deployment, when the cost of fixing them is 10x higher.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Braintrust provides a shared eval runner with per-project metric configs, versioned datasets, and CI/CD integrations, closely matching the shared infra plus product configs pattern described here.
- Promptfoo supports per-project eval configs with pluggable metrics and LLM-as-judge rubrics, used by teams running eval suites for multiple LLM products from a single codebase.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you handle the case where the code assistant's pass@k metric requires executing untrusted generated code at scale?
Sandboxed execution environments (containers or lightweight VMs) per test case, with resource caps on CPU, memory, and network. Pre-built dependency images per language. Kill after timeout. Log stdout/stderr for debugging failures. The eval engine needs an execution backend that is isolated from the runner itself.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Building one monolithic rubric for all three products. A rubric calibrated for summarization faithfulness is meaningless for code correctness, and comparing raw scores across products with different rubrics leads to misallocated engineering effort.
60 second bullets to scan on the way to the call.
Name the shared eval runner architecture with product-specific config bundles
List the four shared infrastructure components: versioned storage, execution engine, results DB, CI/CD gates
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.