Design shadow mode evaluation for a planned model swap from Sonnet to a smaller model

Short answer·Hard·4.0 · 0·~3 min·Asked atFreshworks Paytm Yellow Ai

Attempt it

You want to migrate a chat assistant from Claude Sonnet to a smaller, cheaper model. Describe how you would run a shadow mode evaluation for two weeks before any user visible change, what you would measure, and the failure modes to watch for.

Free · 2 AI evals / day

Shadow mode is the standard way to test a candidate model against the production model on real traffic without exposing users to it. Fork the call site so every sampled production request fires twice. Sonnet's response goes to the user. The candidate runs in parallel and its response is logged, never served. Both spans share a trace id, so per-request comparison is a join. A 10 to 25 percent sample rate gives enough volume in two weeks for statistical significance on most metrics without doubling the LLM bill. Very high-traffic systems can sample lower. Score both responses as a pair with an LLM-as-judge on faithfulness, helpfulness, and task rubrics. Track output length, refusal rate, structured-output validity (JSON parses, tool-call arguments match the schema), latency p50 and p99, and cost per request. Slice every metric by tenant tier, intent, language, and other user-population axes. Smaller models fail differently. Format drift is common: the candidate returns valid JSON less reliably, silently breaking downstream parsers. Tool-call argument shapes may differ subtly. Refusals shift in either direction. The judge may miss a different hallucination pattern, so spot-check disagreements manually. End with a documented go no go that includes worst-cohort delta, not just aggregate.

**Architecture.** Fork the call site behind a feature flag. Every sampled production request fires both Sonnet (response served to user) and the candidate model (response logged, never served). Share a trace id across both spans so paired analysis is a join. Sample at 10-25 percent for cost containment (lower for very high traffic). **Two-week duration** gives roughly 200,000-500,000 shadow comparisons at typical SaaS volumes: enough for paired tests on the main metrics and for cohort-level volume on minority intents. **Measurement (paired wherever possible).** LLM-as-judge scores on faithfulness to retrieved context, helpfulness on the task, tone, and any domain rubric. Format-validity rates (JSON parses, tool-call arguments match schema). Refusal rates and refusal-correctness. Output-length distributions (smaller models are often verbose or terse in a different shape than the baseline). Latency p50, p95, p99. Cost-per-request. Tool-call argument-shape agreement for agent flows. **Cohort slicing.** Tenant tier (free, pro, enterprise), intent or route, input language, input length bucket, presence of retrieved context, presence of tool calls. Aggregate metrics hide cohort regressions. Report and gate on per-cohort deltas. **Failure modes specific to smaller-model swaps.** Format drift (JSON validity drops by a few percentage points and downstream parsers fail silently). Confident-but-wrong outputs with a different hallucination pattern that the judge prompt does not catch. Refusal shift in either direction (the new model refuses more or less, which is a UX regression either way). Tool-call argument mismatches in agent workflows. Long-context degradation that does not appear on short-prompt examples. **Go-no-go.** Documented decision gate at end of week two.

TL;DR

Fork the call so Sonnet serves the user and the candidate runs in parallel. Sample 10-25 percent to bound cost, score paired metrics by cohort, and watch for format drift, tool-call mismatch, and refusal shift.

Memory aid

Easy to grasp

Imagine you want to swap the chef at a busy restaurant for a faster, cheaper chef. You would never just put the new chef on the line during dinner service. Instead, for two weeks the old chef keeps cooking the actual plates going to customers. The new chef cooks the same orders in a back kitchen at the same time. Nobody eats the back-kitchen food, but a panel of tasters scores both versions side by side. You also check that the new chef plates the food correctly, refuses the right special requests, and works at the right speed. At the end, you decide based not just on average score but on the worst category of dish. If the new chef is great at appetizers but terrible at desserts, the swap is not safe.

Key concepts

LLMOps

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Shadow-mode evaluation is the most informative pre-launch test an LLM team can run. It exercises the candidate on the actual production input distribution rather than a curated golden set. Two weeks of shadow at moderate sample rate produces hundreds of thousands of paired comparisons on real traffic. That is enough to detect even small regressions and to identify which user cohorts are affected.

The design pattern is straightforward: fire both models on every sampled request, serve only the primary, log both. The discipline is in the four less-obvious choices: where in the call stack to fork, how to sample, what to measure and slice by, and how to write a go no go decision that the team still applies at week two.

Architecture: fork late, run async, cap costs

The fork point matters more than most teams realize. Suppose you fork too early: before retrieval, before tool resolution, or before the final system prompt is assembled. Then the candidate runs against a slightly different prompt than the primary. Now every per-request difference includes a confounder, and the paired analysis is contaminated. Fork late, immediately before the model call, so the candidate sees the exact same fully-resolved prompt, retrieval results, and tool definitions as the primary.

Run the candidate asynchronously. The primary call must not wait for the candidate to complete. User-visible latency depends only on the primary path. Implement it as a background task that fires the candidate call and writes the result to the trace store with the shared trace id. Failures on the candidate path are logged but never propagated to the user. A timeout on the candidate (e.g., 30 seconds) prevents a stuck candidate call from leaking resources.

Budget caps on the shadow path are non-negotiable. A buggy candidate that consumes 10x the tokens per request will inflate the LLM bill during the shadow phase. Configure the gateway to enforce a generous per-request cost cap on the shadow side (maybe 3x the typical baseline cost), so a single anomalous request does not run up cost. Add a daily total cap on shadow spend.

Feature-flag the shadow ramp. Start at 1 percent for the first day to validate the wiring and watch for any cost or latency surprises. Ramp to 10 to 25 percent for the bulk of the two weeks once the wiring is proven.

Measurement: paired metrics, sliced by cohort, audited with humans

Failure modes specific to swaps to a smaller model

Go-no-go protocol

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Langfuse and LangSmith both ship dataset-runs and paired-comparison primitives that make shadow analysis a configuration choice rather than custom orchestration.
LiteLLM Proxy supports shadow routing where every call fires a primary and a configurable shadow model in parallel with budget caps and per-tenant routing.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat if the candidate fails on one cohort but you still want to ship?

Route by cohort. Keep Sonnet on the regressing cohort and ship the candidate on the others, behind a tenant or intent flag at the gateway. Revisit when the candidate prompt is tuned.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Comparing only aggregate judge scores. The candidate can win on average while quietly regressing on an important cohort (enterprise users, long contexts, tool-using flows). The swap then breaks the wrong customers in production.

60 second bullets to scan on the way to the call.

Dual-fire architecture with primary served and candidate logged
Sample rate (10-25 percent) and rationale for cost containment

Primary sources. Browse if you want the original framing.