You've added a new reranker and your offline eval set shows it improves retrieval quality. Before shipping it to all users, you need to prove the gain is real in production. Describe how you would run this experiment, what you would measure, and what would convince you to roll out.
Offline wins don't guarantee live wins; gate the launch on a randomized A/B test with one pre-committed primary metric, latency and cost guardrails, run to significance.
Imagine you tweak a cookie recipe and a few taste-testers in your kitchen love it. That's promising, but your kitchen isn't the whole bakery's customer base. Before you switch every batch, you run a fair trial: half the customers get the old cookie, half get the new one, and you hand them out at random so the two groups are alike. You decide before you start what counts as success — say, more people coming back for seconds — and what you won't tolerate, like the new cookie taking twice as long to bake. You wait until enough people have tried both to be sure it's not luck, instead of declaring victory after three happy customers. Only if the new cookie clearly wins on the thing you chose, without breaking the rules you set, do you switch the whole bakery over.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
2–4 min · Everything important, quickly.
5-6 min: why offline doesn't equal live, randomized A/B design and unit choice, the pre-committed primary metric, latency/cost guardrails, powering the test, and the no-peeking discipline.
Real products, models, and research that use this idea.
- A support-bot team's new cross-encoder reranker won 4 points on their offline set but moved live thumbs-up by under a point while adding 120ms p95 — the guardrail killed the launch.
- Search and recommendation teams at large platforms gate every ranking change behind a powered A/B test with pre-registered primary and guardrail metrics.
- Statistics engines like those behind experimentation platforms expose sequential / always-valid p-values precisely so teams can monitor without the peeking penalty.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does peeking at a running A/B test and stopping early inflate the false-positive rate?
QWhen would interleaving beat a classic A/B test for a ranking change?
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Shipping on the offline win alone, or peeking at the live test and stopping the moment it looks good — both turn an experiment into a way to confirm what you hoped.
60-second night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.