Offline eval scores 0.9 but users report frequent wrong answers: why the gap, and what do you do?

A high offline score and frequent live failures are not a contradiction — they usually mean the eval set does not look like real traffic. This is distribution shift. Most offline eval sets are clean and narrow. The questions are hand-written or sampled from a tidy slice of popular topics: well-formed, no typos, no follow-up context, nothing out of scope. Real users are messier. They send ambiguous, misspelled, multi-turn, long-tail, and sometimes adversarial questions the eval set never represents. So a 0.9 measures the easy, in-distribution questions, not the ones that actually fail. The set can also be stale — the documents moved on since it was written — or contaminated, because whoever wrote the questions looked at the source documents, making retrieval look easier than it really is. The fix is to make evaluation representative, not to keep tuning against a bad set. Mine real production queries, especially the failures users flag with thumbs-down or escalations, and fold them into the golden set. Separate retrieval metrics from generation metrics so you can see which half is broken. And add live signals a static set can never give you, like user feedback and sampled faithfulness checks.

**The score isn't wrong, it's measuring the wrong population.** The root cause is almost always distribution shift between the offline eval set and live query traffic. Offline sets tend to be built from a clean, narrow slice: well-formed questions on popular topics, no typos, no multi-turn coreference, nothing out of scope. Real users send ambiguous, misspelled, long-tail, multi-turn, and adversarial queries that the set never represents. A 0.9 there tells you the system handles easy in-distribution questions; it says nothing about the messy ones that generate the support tickets. Two failure modes compound this. The set goes stale as the corpus changes. And it gets contaminated when questions are written by looking at the documents, which makes retrieval look far easier than it is in the wild. The fix is to align the measurement with production rather than tune harder against an unrepresentative set. Mine real production queries — especially the failures surfaced by thumbs-down, escalations, and abandoned sessions. Fold them into the golden set so it tracks the live distribution and refreshes as traffic drifts. Then localize. Split retrieval metrics (recall@k, hit rate) from generation metrics (faithfulness, answer correctness). That tells you whether the retriever is missing the chunk or the generator is misusing a chunk it had. Finally, add online signals offline evals cannot give you: live faithfulness sampling, structured user feedback, and A/B comparisons on real traffic. The explicit anti-pattern is continuing to optimize the 0.9 — that just overfits to questions nobody is actually failing on.