Describe a rolling baseline drift detector built on judge scores

A **rolling-baseline drift detector** has two moving windows of judge scores: a short window (recent traffic) and a long window (the baseline). You compare the short window to the long window and alert when they look meaningfully different. The short window is typically the last 1 to 24 hours of traffic. The long window is the last 7 to 30 days, often excluding the very recent traffic so the short window does not pollute the baseline. For each window you compute summary statistics: median (p50) score, tail (p95), and standard deviation. An alert fires when the short window's median drops more than M standard deviations below the long baseline, or by more than a configured absolute delta (say 5 percentage points). The standard-deviation form scales with how noisy the metric naturally is. Why not a fixed threshold? Judge scores drift as models, prompts, and user traffic change. A fixed cutoff like 'alert if quality drops below 0.85' eventually misfires: either it fires every day because seasonal traffic shifts the median, or it never fires because the baseline has slowly improved past it. Rolling baselines move with the system.

**Rolling-baseline drift detection** runs two windows over the same judge-score stream and compares them. - Short window: last 1 to 24 hours of traffic. Captures 'right now'. - Long baseline: last 7 to 30 days, optionally excluding the short window so it does not contaminate the baseline. For each window, compute the relevant percentiles, usually p50 and p95, plus the baseline standard deviation. Alert when the short window's p50 falls below the long baseline's p50 by more than M standard deviations (M typically 2 or 3), or by a configurable absolute delta as a floor (so noisy metrics with tiny variance do not flag on trivial swings). The structural argument over fixed thresholds: judge scores are non-stationary. The model behind the judge gets updated, the prompts get tweaked, the user mix shifts, the baseline drifts. A static 'alert if score < 0.85' either floods the channel during natural dips (false positives) or stops firing once the baseline silently improves past it (false negatives). The rolling baseline adapts: alerts mean today is genuinely worse than recent history, not worse than an aspirational number set six months ago. In practice you also segment: rolling baselines per route, per user tier, per model version. A drop in one segment can hide in the global average; per-segment baselines surface it.