Predict what slice-level eval reveals about an 85% faithful RAG system suspected to fail on multi-hop queries

Attempt it

A RAG system reports 85% overall faithfulness across all queries. A team member suspects it fails specifically on multi-hop queries (questions requiring synthesis of information from multiple documents). The team decides to disaggregate the eval set by query complexity:

- Single-hop queries: 400 examples
- Multi-hop queries: 100 examples

If the suspicion is correct and multi-hop faithfulness is significantly lower, what would the slice-level results most likely show?

Quick reference

TL;DR

The 85% mean hides the failure. Single-hop sits near 95% and multi-hop near 45%, a 50-point gap, because the 400 easy queries outvote the 100 hard ones in the weighted average.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a school reports an 85% average pass rate and everyone relaxes. But the school has 400 easy-test students and only 100 hard-test students. The easy group scored 95%, the hard group scored 45%, and the big easy group dragged the average up to 85%. The single number looked healthy while almost half of one group was failing. Faithfulness in a RAG system works the same way. Faithfulness measures whether an answer's claims are actually backed by the retrieved documents. If you only look at the overall 85%, the 400 simple single-hop questions outvote the 100 hard multi-hop ones, so a severe multi-hop failure stays invisible. The fix is to grade each group separately instead of trusting one blended score.

Key concepts

Slice-level metrics Stratification Faithfulness Simpson's paradox

Quick scan · 2–4 min

Everything important, quickly.

5 min: solve the weighted average for the hidden single-hop slice, explain why volume imbalance lets the mean mask a severe multi-hop failure, then cover stratified reporting and worst-slice gating.

Real-world examples

Real products, models, and research that use this idea.

RAGAS lets you tag queries by type and compute faithfulness per slice, surfacing multi-hop gaps that the dataset mean hides.
LangSmith supports metadata-based slicing so eval dashboards can break faithfulness out by query complexity and document source.
TruLens groundedness scores can be grouped per query category, exposing slices where retrieval fails to support synthesis.
Chatbot Arena style leaderboards report per-category Elo precisely because a single aggregate hides where a model is weak.
Promptfoo and DeepEval both expose per-tag assertion pass rates so teams gate releases on the worst slice rather than the mean.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow confident can you be that multi-hop is 45% given only 100 examples?

Sign in or go Pro to see the approach for this follow-up.

QHow does this connect to Simpson's paradox, and when could the aggregate actually reverse a per-slice conclusion?

Sign in or go Pro to see the approach for this follow-up.

QWhat slicing dimensions beyond query complexity should a RAG eval stratify on?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.