Why is reporting only mean eval score dangerous for LLM quality tracking?

**Slice-level evaluation** means breaking your eval score down by segment instead of reporting one aggregate number. Segments might be query type (medical, coding, casual), user cohort (free vs paid), language, or difficulty band. The danger with a single mean is that it averages across all those segments at once. A mean is just a weighted sum, so a big drop on a small slice can be cancelled out by a small gain on a large slice. The headline number stays flat while a real failure hides underneath. The classic example: medical queries are 5 percent of your traffic but the highest stakes. A new model version drops medical accuracy by 5 points but gains 5 points on casual chat, which is 60 percent of traffic. Your overall mean goes UP. You ship, and the regression on the queries that matter most goes live. The fix is to report the metric per slice, every eval pass. Each segment gets its own score and its own trend line. A regression on any important slice blocks the release even when the aggregate looks fine. The other options are wrong. Means are not statistically biased estimators, computing the mean is not the expensive part, and length conflation is a judge-prompt problem, not an aggregation problem.

**Slice-level evaluation** disaggregates the eval metric by segment instead of collapsing everything into one mean. The core risk is mathematical. A mean is a traffic-weighted average. So a regression on a low-volume, high-stakes slice can be fully masked by a gain on a high-volume, easy slice. The headline number stays flat or even rises while an important sub-population silently breaks. The canonical failure: medical queries are 5 percent of traffic. A new version drops them 5 points but gains 5 points on casual chat at 60 percent of traffic. The aggregate climbs. You ship the regression. The fix is to define slices up front and track each one. Common axes are query type, user cohort, input language, and difficulty band. Each slice gets its own score and trend. A regression on any guarded slice blocks the release, independent of the aggregate. This also defends against Goodhart's law. Teams optimising a single number trade away minority-slice quality for majority-slice gains without realising it. Per-slice gates make that trade visible and stop it. The wrong options. Means are not biased estimators needing correction. Compute cost lives in the eval calls, not the averaging. Length conflation is a judge-prompt issue, not an aggregation issue.