How online sampled evals turn into a continuous quality signal in production

**CI evaluation** runs a fixed test set (the golden set) against your prompt or model every time it changes. It catches regressions on cases you already know about. **Online sampled evaluation** runs a quality check on actual production traffic in real time, on a small percent of requests, so you also catch surprises. The loop is simple. On 1 to 5 percent of production requests, the system makes a second LLM call to a judge model. The judge sees the input, the retrieved context if any, and the response, and scores the response on faithfulness, helpfulness, or whatever rubric matters for the task. The score is written back as an attribute on the trace and aggregated on a dashboard. **Why this catches things CI misses.** A CI golden set is built from past examples. It cannot represent traffic that has not happened yet. A new user cohort, a new product feature, or a single big customer onboarding can send traffic shaped differently from anything the golden set covered. The prompt looks fine on the golden set but starts misbehaving on the new traffic. Online evals see the shift in real time; CI never sees it. The cost is one extra LLM call per sampled request, billed at the judge model's rate. Most teams use a smaller cheaper judge model and accept that the judge is slightly biased, calibrating it against human-labeled samples periodically.

**The loop.** A wrapper around every LLM call (often the gateway, sometimes the observability SDK) samples 1 to 5 percent of production requests, makes a second LLM call to a judge model with a rubric prompt, gets back a structured score (0 to 1 numeric, or a label like 'pass / fail / partial'), and writes that score back as a span attribute on the original request's trace. Langfuse, LangSmith, Phoenix, Helicone, and Datadog LLM Observability all ship this pattern as a first-class feature. **What the dashboard shows.** Rolling baseline (last 24 hours, last 7 days) of the judge score, broken down by prompt version, model, tenant, intent, or any other span attribute the team finds useful. Alerts fire when the rolling baseline drops below threshold or when a specific slice (one tenant, one intent) regresses sharply. **What this catches that CI does not.** The CI golden set is by construction backward-looking. It encodes what the team thought to test as of last release. Production traffic continuously drifts: a marketing push brings a new user cohort, a new product surface routes new question types through the same prompt, a single enterprise onboarding sends 5x volume of one specific intent, a new attack pattern in the wild starts probing the system. Each of these can regress quality without changing the golden-set score. The online judge sees the score on the new traffic immediately. **The cost equation.** Each sampled request adds one judge LLM call. At 5 percent sampling and a Haiku-tier judge, the cost overhead is usually 1 to 3 percent of the production LLM bill. Teams trade sampling rate for cost: 1 percent for very high volume, 5 to 10 percent for lower volume where statistical power matters more. The judge model is almost always a smaller and cheaper model than the production model; the bias from using a smaller judge is calibrated periodically against human-labeled samples on the same traffic.