Design shadow mode evaluation for a planned model swap from Sonnet to a smaller model

Shadow mode is the standard way to evaluate a candidate model against the production model on real traffic without exposing users to the candidate. The architecture is straightforward and the discipline is in what you measure and how you decide. Fork the call site so every (sampled) production request fires twice. The primary call to Sonnet serves the response to the user as usual. The shadow call to the candidate model runs in parallel; its response is logged but never served. Both spans share the same trace id so any per-request comparison is a join on that id. Sampling controls cost. A 10 to 25 percent sample rate gives enough volume in two weeks to reach statistical significance on most metrics without doubling the LLM bill. For very high traffic systems you can sample even lower. Measurement is paired wherever possible. For each shadowed request, score both responses with an LLM-as-judge on faithfulness, helpfulness, and any task-specific rubric. Track output-length distributions, refusal rates, structured-output validity (JSON parses cleanly, tool-call arguments match the schema), latency p50 and p99, and cost per request. Slice every metric by tenant tier, by intent, by language, and by any other axis that distinguishes user populations. The failure modes to watch for with a smaller model are different from the ones for a frontier to frontier swap. Format drift is common: the candidate returns valid JSON less reliably, and parsers downstream silently break. Tool-call argument shapes may differ in subtle ways that break agent workflows. Refusal behavior shifts in either direction. The judge prompt may not catch a different hallucination pattern, so spot-check a sample of disagreements manually. End the shadow phase with a documented go no go that includes worst-cohort delta, not just aggregate. A candidate that wins on average but regresses on the enterprise-tier cohort is not a safe swap.

**Architecture.** Fork the call site behind a feature flag. Every sampled production request fires both Sonnet (response served to user) and the candidate model (response logged, never served). Share a trace id across both spans so paired analysis is a join. Sample at 10-25 percent for cost containment; lower for very high traffic. **Two-week duration** gives roughly 200,000-500,000 shadow comparisons at typical SaaS volumes, which is enough for paired tests on the main metrics and for sufficient cohort-level volume on minority intents. **Measurement (paired wherever possible).** LLM-as-judge scores on faithfulness to retrieved context, helpfulness on the task, tone, and any domain rubric. Format-validity rates (JSON parses, tool-call arguments match schema). Refusal rates and refusal-correctness. Output-length distributions (smaller models often verbose or terse in different shape than the baseline). Latency p50, p95, p99. Cost-per-request. Tool-call argument-shape agreement for agent flows. **Cohort slicing.** Tenant tier (free, pro, enterprise), intent or route, input language, input length bucket, presence of retrieved context, presence of tool calls. Aggregate metrics hide cohort regressions; report and gate on per-cohort deltas. **Failure modes specific to smaller-model swaps.** Format drift (JSON validity drops by a few percentage points and downstream parsers fail silently). Confident-but-wrong outputs with a different hallucination pattern that the judge prompt does not catch. Refusal shift in either direction (the new model refuses more or less, which is a UX regression either way). Tool-call argument mismatches in agent workflows. Long-context degradation that does not appear on short-prompt examples. **Go-no-go.** Documented decision gate at end of week two.