Order the rollout of a semantic search relaunch over a legacy keyword engine
- 1Decommission the legacy only path once metrics hold at full traffic
- 2Run shadow traffic: keep serving keyword results to users while logging what the vector path would have returned, and compare
- 3Ramp the cohort while watching click through, zero result rate, and latency dashboards
- 4Ship hybrid retrieval (keyword plus vector) behind a feature flag to a small A/B cohort
- 5Build an offline eval set from real logged queries with judged relevance, and baseline the legacy keyword engine on it
- 6Embed the full document corpus offline and backfill the vector store
Index, baseline, shadow, hybrid behind a flag, ramp, decommission. Every step de-risks the next, and teardown is always last.
Imagine replacing the kitchen in a busy restaurant while it stays open. First, you build and stock the new kitchen in the back. Then you cook the same dishes in both kitchens and compare them blind. Once the food matches or beats the old kitchen, you let a few tables order from the new kitchen as a test. If the reviews are good, you slowly send more tables there. Only after everyone is happy with the new kitchen for a long time do you tear down the old one. The mistakes people make in rollouts are skipping steps. You cannot compare kitchens without building the new one first. You cannot send tables there without comparing first. You do not tear down the old one until the new one has earned trust.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Replacing a retrieval engine that users have learned to query is one of the highest-risk migrations in a production system. Users have built habits around the quirks of the legacy engine. Their queries fit its tokenization, and their expectations match its recall. Their happiness depends on consistency. Switch to a new engine all at once, and a meaningful share of users will see the change as a regression. This happens even if the new engine wins on average.
A six-step playbook is the conservative pattern that has emerged from real relaunches at search-heavy products. Each step measures something the next step needs and de-risks the broader rollout. Skipping any step trades a known cost (engineering time) for an unknown one (production surprises).
Step 1: index the corpus
The index has to exist before anything else can run. Embedding the full document corpus into vectors and loading them into the vector store is the foundation everything else builds on.
The non-obvious work in this step is picking the chunking strategy, embedding model, and dimensionality. Changing any of those later forces a full reindex and invalidates downstream measurements. Standard choices: sentence-aware chunking at roughly 500 tokens, an embedding model in the 768 to 3072 dimension range (depending on quality and cost tradeoffs), and a vector store that supports filtering and hybrid search natively.
Freshness has to be planned here too. A static one-shot index is fine for a research demo but useless for production. The relaunch plan has to include an incremental update pipeline that keeps the vector index synced with the source of truth. Ideally, it consumes the same change events the legacy engine uses.
Skipping this step means starting downstream work against a corpus that does not exist yet. That sounds absurd, but it happens regularly. Teams build eval sets against synthetic data because the real index is not ready. The eval set built against the wrong corpus has to be rebuilt later, doubling the work.
Index sizing is a rough calculation worth doing before you commit. For a 10M-document corpus chunked at roughly 5 chunks per doc, you store 50M vectors. At 1024 dimensions and float32, raw vectors alone consume before ANN graph overhead (typically 1.5x on HNSW). Quantization to int8 cuts this fourfold with a small recall penalty, and is the standard production trade. Qdrant and Weaviate support 8-bit quantization. pgvector supports half-precision and binary quantization. Cost the embed pass too. Embedding 50M chunks at USD 0.02 per million tokens with 500 tokens per chunk costs about USD 500 in one shot. The incremental delta pipeline is where the steady-state cost comes from.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Search teams commonly run retrieval relaunches in this six-step shape, with names varying for shadow and ramp
- Elastic and OpenSearch support hybrid search (keyword plus vector) for production semantic search
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat if shadow traffic shows the vector path is slower than legacy?
Latency regressions surface here, before users feel them. Investigate prefill cost, ANN parameters (ef, nprobe), and index sharding. If the gap is fundamental, decide whether to ship anyway (if quality gains justify it) or invest in serving optimizations before continuing.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Shipping vector search to users before running shadow traffic. Recall then drops on a class of queries the offline eval did not cover, and you find out in production.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.