Your team reports a 4.1/5.0 average helpfulness score across all queries, but users in regulated industries are complaining about inaccurate answers. How do you redesign the evaluation strategy using slice level metrics to diagnose and track this problem?
Stop reporting one mean. Slice the eval set by intent, cohort, difficulty, and risk; report per-slice scores with adequate sample sizes; gate release on the high-risk slices, not the average.
Imagine a school that brags its students average 85 percent. That number hides the fact that the advanced-physics class is failing badly while the easy classes pull the average up. A single number lets a small struggling group disappear into a big happy crowd. The fix is to report a grade for each class separately, especially the hard and high-stakes ones. You also need enough students in each class for the grade to mean anything, and you decide which classes are allowed to drag down the school. For an LLM, the classes are slices: regulated-industry questions, hard questions, each user group. You grade each slice on its own, and you refuse to ship if a critical slice gets worse, even when the overall average looks fine.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Slice-level evaluation is the practice of reporting quality per meaningful subgroup of inputs rather than as a single global average. It exists because the mean is a lossy compression of a distribution: a system can post a flat 4.1 out of 5 while a small, high-stakes slice quietly collapses to 3.2, and the happy majority hides the damage.
This question is a staff-level measurement-system design problem disguised as a metric question. The interviewer wants to see four things: a principled way to choose slices, a defensible per-slice sample-size argument, control of false alarms when you test many slices at once, and a release gate that ranks slices by business risk rather than traffic volume. The deep dive walks each in turn and closes with the dashboard and worked numbers.
Why the mean hides the regression
A global mean is an average weighted by traffic volume. If 95 percent of queries are casual and score 4.2, and 5 percent are regulated and score 3.2, the blended mean is about 4.15. The regulated failure is mathematically invisible: the casual majority dominates the weighted sum, so even a severe drop in the small slice barely moves the headline number.
The deeper issue is that a mean answers the wrong question. Stakeholders do not experience an average; each user experiences their own slice. A regulated customer feels the 3.2, not the 4.15. The eval's job is to predict the experience of each population that matters, which means you must stop collapsing populations together.
The fix is structural, not a better metric. You report a vector of per-slice scores and you commit, in advance, to which slices you refuse to let aggregation hide. That commitment is the heart of the design.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangSmith and Braintrust let you tag eval examples and view per-slice score breakdowns rather than only a single aggregate metric.
- RAGAS lets teams stratify faithfulness and answer-relevance scores per query type so a weak regulated-domain slice does not vanish into the mean.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you decide the minimum sample size for a slice that must gate a release?
Work backward from the minimum detectable regression. Pick the smallest score drop that matters, then size the slice so its bootstrap confidence interval is narrower than that drop at your chosen confidence. This usually lands at a few hundred examples and forces oversampling of rare slices.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Slicing only by traffic volume and reporting per-slice means without sample sizes or confidence intervals, so tiny high-risk slices stay invisible and noise reads as a regression.
60 second bullets to scan on the way to the call.
The four slicing axes: intent, cohort, difficulty, risk
Why a global mean hides a small high-risk regression
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.