Build an eval framework from scratch for three LLM products (support bot, code assistant, summarizer). Cover architecture, shared infra, per-product customization, and six-month failure modes.
Build an eval framework from scratch for three LLM products (support bot, code assistant, summarizer). Cover architecture, shared infra, per-product customization, and six-month failure modes.
Shared eval runner with per-product configs, versioned storage, results DB, CI gates. Four failure modes: rubric drift, judge deprecation, golden set staleness, cross-product comparison trap.
Imagine three restaurants in the same building sharing one kitchen inspection system. Each restaurant serves different food, so the inspection checklist is different for each: the sushi place checks fish freshness, the bakery checks oven temperatures, the salad bar checks produce sourcing. The shared part is the inspection schedule, the filing system for reports, and the alarm that rings when a score drops. But if the sushi menu changes and nobody updates the checklist, inspections pass while customers get sick. If the inspector retires and nobody trains a replacement, all three restaurants lose their reviewer on the same day. If the filing system only stores last month's scores, nobody notices a slow decline. And if the building manager compares the sushi score to the bakery score and concludes the bakery is worse, that comparison is meaningless because the checklists measure different things.
Detailed answer & concept explanation~9 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Open with the architecture: shared eval runner loading product-specific config bundles. Walk the four shared infra components (versioned storage, execution engine, results DB, CI/CD gates). Then walk per-product customization showing why metrics diverge (pass@k vs ROUGE vs multi-turn). Close with the four failure modes in order: rubric drift, judge deprecation, golden set staleness, cross-product comparison trap. For each, name the failure, explain why it is predictable, and give the mitigation.
Real products, models, and research that use this idea.
- Braintrust provides a shared eval runner with per-project metric configs, versioned datasets, and CI/CD integrations, closely matching the shared infra plus product configs pattern described here.
- Promptfoo supports per-project eval configs with pluggable metrics and LLM-as-judge rubrics, used by teams running eval suites for multiple LLM products from a single codebase.
- Anthropic's internal eval infrastructure uses product-specific golden sets with quarterly refresh cycles and judge model versioning to avoid the staleness and deprecation failure modes.
- LangSmith provides run comparison dashboards and regression alerting across multiple projects, demonstrating the shared results database and trend monitoring layer.
- Arize Phoenix offers production monitoring with per-product eval traces, enabling teams to detect distribution shift between eval-time and production-time queries.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you handle the case where the code assistant's pass@k metric requires executing untrusted generated code at scale?
QThe support bot team wants to eval multi-turn conversations, but your golden set only has single-turn examples. How do you extend the framework?
QTwo products want to use different judge models. How does that affect the shared infrastructure?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Building one monolithic rubric for all three products. A rubric calibrated for summarization faithfulness is meaningless for code correctness, and comparing raw scores across products with different rubrics leads to misallocated engineering effort.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.