**Slice-level evaluation** means breaking your eval score down by segment instead of reporting one aggregate number. Segments might be query type (medical, coding, casual), user cohort (free vs paid), language, or difficulty band.
The danger with a single mean is that it averages across all those segments at once. A mean is just a weighted sum, so a big drop on a small slice can be cancelled out by a small gain on a large slice. The headline number stays flat while a real failure hides underneath.
The classic example: medical queries are 5 percent of your traffic but the highest stakes. A new model version drops medical accuracy by 5 points but gains 5 points on casual chat, which is 60 percent of traffic. Your overall mean goes UP. You ship, and the regression on the queries that matter most goes live.
The fix is to report the metric per slice, every eval pass. Each segment gets its own score and its own trend line. A regression on any important slice blocks the release even when the aggregate looks fine.
The other options are wrong. Means are not statistically biased estimators, computing the mean is not the expensive part, and length conflation is a judge-prompt problem, not an aggregation problem.
**Slice-level evaluation** disaggregates the eval metric by segment instead of collapsing everything into one mean.
The core risk is mathematical. A mean is a traffic-weighted average. So a regression on a low-volume, high-stakes slice can be fully masked by a gain on a high-volume, easy slice. The headline number stays flat or even rises while an important sub-population silently breaks.
The canonical failure: medical queries are 5 percent of traffic. A new version drops them 5 points but gains 5 points on casual chat at 60 percent of traffic. The aggregate climbs. You ship the regression.
The fix is to define slices up front and track each one. Common axes are query type, user cohort, input language, and difficulty band. Each slice gets its own score and trend. A regression on any guarded slice blocks the release, independent of the aggregate.
This also defends against Goodhart's law. Teams optimising a single number trade away minority-slice quality for majority-slice gains without realising it. Per-slice gates make that trade visible and stop it.
The wrong options. Means are not biased estimators needing correction. Compute cost lives in the eval calls, not the averaging. Length conflation is a judge-prompt issue, not an aggregation issue.
Quick reference
TL;DR
A single mean blends slices, so a regression on a small high-stakes segment can be cancelled by a gain on high-volume easy queries. Slice-level reporting surfaces hidden failures.
Imagine a school reports one average grade for the whole student body. The number looks healthy at 82 percent. But hidden inside it, the top students climbed from 90 to 95 while the struggling students dropped from 70 to 60. The average barely moved, so nobody noticed the kids who needed help fell off a cliff. LLM eval works the same way. If you report one mean across all user queries, a sharp drop on medical questions can be perfectly masked by a small bump on casual chit-chat. The headline number stays flat and green. To catch the real problem you have to break the grade down by group: medical queries here, coding queries there, rare languages over there. Only then do you see which slice quietly broke.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Very small total eval set where any slice would have too few examples Per-slice estimates become noise. Grow the set or report aggregate with explicit caveats first.
Genuinely homogeneous tasks with one query type and one cohort There is nothing to disaggregate. A single score already describes the whole population.
Exploratory smoke checks during early prototyping before slices are defined A quick aggregate catches gross breakage. Add slicing once the product has real segments.
Reporting a single headline number to non-technical stakeholders An aggregate communicates direction, but keep the slice breakdown as the decision artifact behind it.
Quick scan · 2–4 min
Everything important, quickly.
4 min: why a mean is a weighted sum that masks slice regressions, the medical vs casual example, choosing slice axes, per-slice gating, multiple-comparisons control, and the Goodhart and fairness parallels.
Real-world examples
Real products, models, and research that use this idea.
1LangSmith and Braintrust both let you tag eval examples and chart scores per slice, not just one aggregate.
2OpenAI Evals and Promptfoo support per-tag and per-category breakdowns so regressions surface by segment.
3Fairness audits on hiring and lending models report disaggregated group metrics precisely because the aggregate hides harm.
4RAG teams using RAGAS routinely slice faithfulness and context recall by document source and query intent.
5Chatbot Arena reports category leaderboards (coding, math, longer query) so a model strong overall but weak on code is visible.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you decide which slices to guard and how many examples each needs?
Sign in or go Pro to see the approach for this follow-up.
QIf you test twenty slices each pass, how do you avoid false regression alarms?
Sign in or go Pro to see the approach for this follow-up.
QHow does slice-level eval connect to fairness and Goodhart's law?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Shipping on a flat or improving aggregate score without disaggregating. A gain on high-volume easy queries can fully mask a severe regression on a small high-stakes slice.