Predict what slice level eval reveals about an 85% faithful RAG system suspected to fail on multi-hop queries
A RAG system reports 85% overall faithfulness across all queries. A team member suspects it fails specifically on multi-hop queries (questions requiring synthesis of information from multiple documents). The team decides to disaggregate the eval set by query complexity: - Single hop queries: 400 examples - Multi-hop queries: 100 examples If the suspicion is correct and multi-hop faithfulness is significantly lower, what would the slice level results most likely show?
The 85% mean hides the failure. Single-hop sits near 95% and multi-hop near 45%, a 50-point gap, because the 400 easy queries outvote the 100 hard ones in the weighted average.
Imagine a school reports an 85% average pass rate and everyone relaxes. But the school has 400 easy-test students and only 100 hard-test students. The easy group scored 95%, the hard group scored 45%, and the big easy group dragged the average up to 85%. The single number looked healthy while almost half of one group was failing. Faithfulness in a RAG system works the same way. Faithfulness measures whether an answer's claims are actually backed by the retrieved documents. If you only look at the overall 85%, the 400 simple single-hop questions outvote the 100 hard multi-hop ones, so a severe multi-hop failure stays invisible. The fix is to grade each group separately instead of trusting one blended score.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Faithfulness measures whether the claims in a generated answer are grounded in the retrieved context. A RAG system reports 85% overall faithfulness, and a teammate suspects multi-hop queries, which require synthesizing information across multiple documents, fail far more often. The team disaggregates the eval set into 400 single-hop examples and 100 multi-hop examples.
The question is what the slice-level eval will reveal if the suspicion is right. The answer is almost forced by arithmetic: single-hop near 95% and multi-hop near 45%, a 50-percentage-point gap that the 85% headline completely concealed.
This deep dive works the math, explains why the imbalance causes the masking, connects it to Simpson's paradox, sizes the uncertainty on the small slice, and lays out the stratified-reporting discipline that prevents this failure in production. The recurring theme is that an aggregate metric is a modeling choice, not a neutral summary, and choosing it carelessly manufactures false confidence.
Solving the weighted average
The reported 85% is not a property of any single query. It is a volume-weighted mean over two slices of very different sizes. Start from the definition of that mean and solve for the unknown single-hop rate.
The total faithful "mass" across the whole eval set is 500 times 0.85, which equals 425. Think of this as 425 faithful query-units out of 500. Multi-hop, at a suspected 45%, contributes 100 times 0.45, which is 45 units. Everything else must come from single-hop.
That leaves 425 minus 45, which is 380 units, spread across 400 single-hop queries. So single-hop faithfulness is 380 divided by 400, which is 0.95. The slices are therefore about 95% and 45%, a 50-point gap living quietly under an 85% headline.
Notice the move you just made. You inverted the aggregate using one extra fact, the suspected multi-hop rate, plus the known slice sizes. This is the everyday skill behind reading any disaggregated report: given a total and all but one of its parts, the last part is determined. Interviewers like this scenario because it tests whether you can run the weighted mean backward under pressure, not just recite that slicing is good practice. The same inversion lets you sanity-check a vendor's blended benchmark number whenever they publish the slice sizes but bury the per-slice scores.
0.85 = \frac{400 \cdot s + 100 \cdot m}{500} \;\Rightarrow\; s = \frac{425 - 100m}{400}Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- RAGAS lets you tag queries by type and compute faithfulness per slice, surfacing multi-hop gaps that the dataset mean hides.
- LangSmith supports metadata-based slicing so eval dashboards can break faithfulness out by query complexity and document source.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow confident can you be that multi-hop is 45% given only 100 examples?
Compute a binomial confidence interval: at 45% over 100 trials the standard error is about 5 points, so a roughly plus or minus 10-point interval. The gap is clearly real, but the point estimate is fuzzy. Collect more multi-hop examples to tighten it before reporting a precise number.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reading the 85% aggregate as a per-query guarantee. A weighted mean over imbalanced slices can hide a near-total failure on a small but critical slice.
60 second bullets to scan on the way to the call.
The weighted-average identity and how to solve for the hidden slice
Why a high-volume slice dominates the aggregate mean
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.