Build an eval framework from scratch for three LLM products (support bot, code assistant, summarizer). Cover architecture, shared infra, per-product customization, and six-month failure modes.
Build an eval framework from scratch for three LLM products (support bot, code assistant, summarizer). Cover architecture, shared infra, per-product customization, and six-month failure modes.
Free · 2 AI evals / day
An eval framework for three LLM products needs a shared execution layer and per-product customization.
**Shared infrastructure** includes versioned test case storage, a parallel eval execution engine, a results database with trend comparison, and CI/CD hooks that block merges on regression. All three products use the same runner, storage, and alerting.
**Per-product customization** is where each product defines its own golden set, metric stack, rubric, and judge prompts. The code assistant needs `pass@k` for functional correctness. The summarizer needs ROUGE plus faithfulness scoring. The support bot needs multi-turn coherence and escalation detection. Different products, different definitions of quality.
Four failure modes appear within six months. Rubric drift, where products evolve but rubrics stay frozen. Judge model deprecation, where the provider retires the model powering all three eval pipelines. Golden set staleness, where production traffic shifts but the eval set does not. The cross-product comparison trap, where stakeholders compare scores across products with incompatible rubrics.
**Architecture.** A shared eval runner loads product-specific config bundles (golden set, metrics, rubric, judge model, thresholds). The runner is product-agnostic; all product-specific logic lives in the config.
**Shared infra.** Versioned test case storage (git-backed, every run reproducible). Eval execution engine with caching and parallelism. Results database with run comparison, trend dashboards, and regression alerting. CI/CD pre-merge gates that block deployment when scores regress past the threshold.
**Per-product customization.** Code assistant: `pass@k`, style scores, latency. Summarizer: ROUGE, faithfulness (no hallucinated claims), conciseness. Support bot: multi-turn coherence, escalation detection, policy compliance. Each product owns its own LLM-as-judge rubric and version-locked judge prompts.
**Failure modes.** (1) Rubric drift: products evolve faster than rubrics update; scores stay high while users complain. Tie rubric review to the release cycle. (2) Judge model deprecation: provider retires the model, breaking all pipelines simultaneously. Abstract the judge behind a versioned interface with pinned model IDs. (3) Golden set staleness: production distribution shifts, eval set stays frozen. Quarterly refresh with contamination checks. (4) Cross-product comparison trap: stakeholders compare raw scores across different rubrics. Mitigate with product-level dashboards and explicit warnings against cross-product comparison.
Quick reference
TL;DR
Shared eval runner with per-product configs, versioned storage, results DB, CI gates. Four failure modes: rubric drift, judge deprecation, golden set staleness, cross-product comparison trap.
Imagine three restaurants in the same building sharing one kitchen inspection system. Each restaurant serves different food, so the inspection checklist is different for each: the sushi place checks fish freshness, the bakery checks oven temperatures, the salad bar checks produce sourcing. The shared part is the inspection schedule, the filing system for reports, and the alarm that rings when a score drops. But if the sushi menu changes and nobody updates the checklist, inspections pass while customers get sick. If the inspector retires and nobody trains a replacement, all three restaurants lose their reviewer on the same day. If the filing system only stores last month's scores, nobody notices a slow decline. And if the building manager compares the sushi score to the bakery score and concludes the bakery is worse, that comparison is meaningless because the checklists measure different things.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A company shipping a single LLM product with one use case. The multi-product architecture adds unnecessary abstraction when there is only one config to load. A simpler eval script with versioned golden sets and CI hooks is sufficient.
An early prototype where the product definition is still changing weekly. Building a full eval framework before the product stabilizes means the framework's configs, rubrics, and golden sets will be rewritten repeatedly. Start with ad hoc eval scripts and formalize after the product shape settles.
A team that has not yet defined what quality means for their product. The framework assumes each product has a clear metric stack and rubric. Without a quality definition, the framework has nothing to evaluate. Define metrics first, then build infrastructure.
Products that share identical quality criteria and rubrics. The per-product customization layer adds overhead when all products use the same metrics. A single eval config with shared rubrics is simpler and avoids the illusion of product-specific tuning.
Quick scan · 2–4 min
Everything important, quickly.
Open with the architecture: shared eval runner loading product-specific config bundles. Walk the four shared infra components (versioned storage, execution engine, results DB, CI/CD gates). Then walk per-product customization showing why metrics diverge (pass@k vs ROUGE vs multi-turn). Close with the four failure modes in order: rubric drift, judge deprecation, golden set staleness, cross-product comparison trap. For each, name the failure, explain why it is predictable, and give the mitigation.
Real-world examples
Real products, models, and research that use this idea.
1Braintrust provides a shared eval runner with per-project metric configs, versioned datasets, and CI/CD integrations, closely matching the shared infra plus product configs pattern described here.
2Promptfoo supports per-project eval configs with pluggable metrics and LLM-as-judge rubrics, used by teams running eval suites for multiple LLM products from a single codebase.
3Anthropic's internal eval infrastructure uses product-specific golden sets with quarterly refresh cycles and judge model versioning to avoid the staleness and deprecation failure modes.
4LangSmith provides run comparison dashboards and regression alerting across multiple projects, demonstrating the shared results database and trend monitoring layer.
5Arize Phoenix offers production monitoring with per-product eval traces, enabling teams to detect distribution shift between eval-time and production-time queries.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you handle the case where the code assistant's pass@k metric requires executing untrusted generated code at scale?
Sign in or go Pro to see the approach for this follow-up.
QThe support bot team wants to eval multi-turn conversations, but your golden set only has single-turn examples. How do you extend the framework?
Sign in or go Pro to see the approach for this follow-up.
QTwo products want to use different judge models. How does that affect the shared infrastructure?
Sign in or go Pro to see the approach for this follow-up.
QHow would you detect that a golden set refresh introduced contamination from the model's training data?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Building one monolithic rubric for all three products. A rubric calibrated for summarization faithfulness is meaningless for code correctness, and comparing raw scores across products with different rubrics leads to misallocated engineering effort.