You've added a new reranker and your offline eval set shows it improves retrieval quality. Before shipping it to all users, you need to prove the gain is real in production. Describe how you would run this experiment, what you would measure, and what would convince you to roll out.
Offline wins don't guarantee live wins; gate the launch on a randomized A/B test with one pre-committed primary metric, latency and cost guardrails, run to significance.
Imagine you tweak a cookie recipe and a few taste-testers in your kitchen love it. That's promising, but your kitchen isn't the whole bakery's customer base. Before you switch every batch, you run a fair trial: half the customers get the old cookie, half get the new one, and you hand them out at random so the two groups are alike. You decide before you start what counts as success — say, more people coming back for seconds — and what you won't tolerate, like the new cookie taking twice as long to bake. You wait until enough people have tried both to be sure it's not luck, instead of declaring victory after three happy customers. Only if the new cookie clearly wins on the thing you chose, without breaking the rules you set, do you switch the whole bakery over.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Every RAG team eventually builds a new component — a reranker, a different embedding model, a chunking change — runs it against a saved eval set, sees a better number, and feels the pull to ship. This question exists to test whether you'll resist that pull and what you'll do instead.
The trap is subtle because the offline number isn't fake. The new reranker really did score higher on that set. The problem is what the set represents. It's a snapshot, curated and small, and the decision in front of you is about all future live traffic. Bridging that gap is an experimentation problem, not a modeling one, and the strongest answers treat it that way: a randomized experiment, a metric chosen before the data arrives, guardrails that protect what you can't afford to break, and the statistical discipline to not fool yourself.
The reranker detail makes the question sharper than a generic 'how do you A/B test' prompt. A reranker buys quality by spending latency and compute, so it's a textbook case where the headline metric and the guardrails point in opposite directions. The interview is really asking whether you'll measure the tradeoff honestly or wave it away with the offline number.
Why the offline win is a hypothesis, not a verdict
An offline eval set does real work. It lets you screen candidates cheaply, catch regressions, and rank ideas without touching users. What it can't do is represent production. It's a fixed, usually hand-curated batch of queries with frozen relevance labels, and it almost always under-samples the long tail — the rare, weird, ambiguous queries where a reranker's behavior actually diverges from the baseline.
There's also leakage risk. Eval sets get tuned on over time; a reranker that looks great may simply be fitting the quirks of that particular set rather than generalizing. And labels go stale: the corpus changes, user intent shifts, and last quarter's 'correct' document may no longer be the best answer.
So the right mental model is that the offline gain is a hypothesis: 'this reranker will improve live outcomes.' A hypothesis earns an experiment, not a launch. Treating the offline number as proof is the single most common failure here, and naming it explicitly is what an interviewer is listening for. The experiment is how you find out whether the hypothesis survives contact with real users.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- A support-bot team's new cross-encoder reranker won 4 points on their offline set but moved live thumbs-up by under a point while adding 120ms p95 — the guardrail killed the launch.
- Search and recommendation teams at large platforms gate every ranking change behind a powered A/B test with pre-registered primary and guardrail metrics.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does peeking at a running A/B test and stopping early inflate the false-positive rate?
Each look is another chance for random noise to cross the significance threshold. Many looks compound that chance, so the effective false-positive rate far exceeds the nominal 5%. Fix the horizon up front, or use a sequential test with always-valid p-values designed for continuous monitoring.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Shipping on the offline win alone, or peeking at the live test and stopping the moment it looks good — both turn an experiment into a way to confirm what you hoped.
60 second bullets to scan on the way to the call.
Why an offline eval gain doesn't guarantee a live gain
Randomized control versus treatment A/B split and the randomization unit
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.