Order the rollout of a semantic search relaunch over a legacy keyword engine
- 1Decommission the legacy only path once metrics hold at full traffic
- 2Run shadow traffic: keep serving keyword results to users while logging what the vector path would have returned, and compare
- 3Ramp the cohort while watching click through, zero result rate, and latency dashboards
- 4Ship hybrid retrieval (keyword plus vector) behind a feature flag to a small A/B cohort
- 5Build an offline eval set from real logged queries with judged relevance, and baseline the legacy keyword engine on it
- 6Embed the full document corpus offline and backfill the vector store
Index, baseline, shadow, hybrid behind a flag, ramp, decommission. Every step de-risks the next, and teardown is always last.
Imagine replacing the kitchen in a busy restaurant while it stays open. First, you build and stock the new kitchen in the back. Then you cook the same dishes in both kitchens and compare them blind. Once the food matches or beats the old kitchen, you let a few tables order from the new kitchen as a test. If the reviews are good, you slowly send more tables there. Only after everyone is happy with the new kitchen for a long time do you actually tear down the old one. The mistakes people make in rollouts are skipping steps. You cannot compare kitchens without building the new one first. You cannot send tables there without comparing first. You do not tear down the old one until the new one has earned trust.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Replacing a retrieval engine that users have learned to query is one of the highest-risk migrations in a production system. Users have built habits around the quirks of the legacy engine. Their queries fit its tokenization, their expectations match its recall, their happiness depends on consistency. Drop a new engine in cold and a meaningful slice of users will perceive the change as a regression even if the new engine wins on average.
The six-step playbook in this question is the conservative pattern that has emerged from real relaunches at search-heavy products. Each step measures something the next step needs and de-risks the broader rollout. Skipping any step trades a known cost (engineering time) for an unknown one (production surprises).
This deep dive walks each step, explains what specifically it measures, names the failure mode of skipping it, and shows how the steps interlock so the relaunch can be defensible at any review checkpoint.
Step 1: index the corpus
The index has to exist before anything else can run. Embedding the full document corpus into vectors and loading them into the vector store is the foundation everything else builds on.
The non-obvious work in this step is picking the chunking strategy, embedding model, and dimensionality, because changing any of those later forces a full reindex and invalidates downstream measurements. Standard choices in 2026: sentence-aware chunking at roughly 500 tokens, an embedding model in the 768 to 3072 dimension range (depending on quality and cost tradeoffs), and a vector store that supports filtering and hybrid search natively.
Freshness has to be planned here too. A static one-shot index is fine for a research demo but useless for production. The relaunch plan has to include an incremental update pipeline that keeps the vector index synced with the source of truth, ideally consuming the same change events the legacy engine uses.
Skipping this step means starting downstream work against a corpus that does not exist yet, which sounds absurd but happens regularly when teams build eval sets against synthetic data because the real index is not ready. The eval set built against the wrong corpus has to be rebuilt later, doubling the work.
Index sizing is a back of the envelope calculation worth doing before commit. For a 10M-document corpus chunked at roughly 5 chunks per doc, you store 50M vectors. At 1024 dimensions and float32, raw vectors alone consume before ANN graph overhead (typically 1.5x on HNSW). Quantization to int8 cuts this fourfold with a small recall penalty, and is the standard production trade in 2026. Pinecone serverless, Weaviate, Qdrant, and pgvector all support int8 quantization natively. Cost the embed pass too: embedding 50M chunks at $0.02 per million tokens with 500 tokens per chunk costs about $500 in one shot, but the incremental delta pipeline is where steady-state cost lives.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Pinterest, Spotify, and Etsy have all published retrieval-relaunch playbooks that follow this six-step shape with names varying for shadow and ramp
- Elastic and OpenSearch ship hybrid search as the default for production semantic-search migrations, not pure vector
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat if shadow traffic shows the vector path is slower than legacy?
Latency regressions surface here, before users feel them. Investigate prefill cost, ANN parameters (ef, nprobe), and index sharding. If the gap is fundamental, decide whether to ship anyway (if quality gains justify it) or invest in serving optimizations before continuing.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Shipping vector search to users before running shadow traffic, then discovering on production that recall has dropped on a class of queries the offline eval did not cover.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.