Order the zero downtime cutover sequence for swapping embedding models under live traffic with rollback safety.

Order steps·Hard·4.0 · 0·~1 min·Asked atHugging Face Mckinsey Together Ai

Attempt it

1After a stable soak period, stop dual writes, archive or drop the old collection, and retire the old model
2Backfill the historical corpus into the new collection in rate limited, checkpointed batches
3Flip read traffic to the new collection behind a feature flag, keeping dual writes running so rollback is instant
4Shadow read: mirror a slice of live queries to the new collection and compare retrieval quality offline
5Start dual writes: every new and updated document is embedded with both models and written to both collections
6Create a new collection for the new model's vectors, with its own dimension, metric, and index parameters

An **embedding model swap** changes the meaning of every vector in the database. The new model produces different numbers, often a different dimension, and crucially the old and new vectors are not comparable. You cannot ship the new model on top of old vectors, and you cannot delete the old vectors first because users are querying them right now. The safe pattern has six steps and the order matters. First, create a separate collection for the new model so old and new live side by side. Second, turn on dual writes so any new document is embedded with both models. Third, backfill the history with the new model, in batches, with checkpoints in case the job fails. Fourth, shadow-test by sending real queries to both and comparing results offline. Fifth, flip reads to the new collection behind a feature flag so you can roll back in seconds. Sixth, after the system has run cleanly on the new model for a while, stop dual writes and remove the old collection. The single most common mistake is starting the backfill before dual writes are on. Any document that gets written during the backfill only lands in the old collection, so the new index is incomplete the moment the backfill finishes.

**Six steps, in this order, because each one preserves a rollback property.** 1. **Provision a new collection** for the new model. It needs its own dimension, distance metric, and index parameters (HNSW `M`, `ef_construction`, or IVF `nlist`). Doing this first means the old collection is untouched and the new one is empty and ready. 2. **Start dual writes.** Every new or updated document is embedded by both models and written to both collections. This must happen before the backfill, or every write during the backfill window is silently lost from the new index. 3. **Backfill the historical corpus** into the new collection in rate-limited, checkpointed batches. Checkpointing matters because the backfill takes hours to days; an unrecoverable failure halfway through wastes the whole run. 4. **Shadow-read.** Mirror a slice of production queries to the new collection and diff retrieval quality offline (recall, MRR, NDCG against a labelled set). This is the gate: if shadow numbers are worse than production, you stop here, not at step five. 5. **Flip reads behind a feature flag,** gradually (1%, 10%, 50%, 100%). Dual writes stay on. The flag is the rollback handle: one toggle returns to the old collection with no data loss because both collections are still receiving writes. 6. **Soak, then retire.** After a stable period (typically one to two weeks), stop dual writes, archive or drop the old collection, and decommission the old model. **The invariant:** at every step between two and six, both collections are complete and correct. Any step can roll back to its predecessor without data loss.

TL;DR

Provision the new collection, start dual writes before backfill, shadow-compare quality, flip reads behind a flag, keep dual writes through soak, then retire the old model.

Memory aid

Easy to grasp

Swapping an embedding model under live traffic is like changing the engine on a car that is already driving. You cannot stop the car. The trick is to mount the new engine alongside the old one, run both for a while, switch the driveshaft over only after you have confirmed the new engine produces the same speed, and keep the old engine bolted on for a few more days in case something feels off. The dangerous step is starting the swap by ripping out the old engine first. Every move in this sequence exists to make sure that if any single step looks wrong, you can fall back to the previous state without losing a single document or breaking a single user query.

Key concepts

Vector Databases

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Embedding-model swaps are deceptively dangerous. The math part (different model, different distribution, sometimes different dim) is small and well understood. The hard part is that the corpus is live, users are querying it continuously, and any moment of inconsistency, either missing documents in the new index or mismatched vectors in the old one, is user-visible as broken search.

The industry-standard cutover sequence is six steps. The order is not a convention; it is the only order that maintains the invariant that both collections are complete and correct at every moment between step two and step six. That invariant is what makes rollback safe.

This deep-dive walks through each step, the failure mode it prevents, and the operational subtleties that turn a textbook sequence into a smooth production cutover.

Why the sequence is not negotiable

The invariant

Between step 2 (dual writes on) and step 6 (soak ends), the system maintains: both collections are complete and correct. Old collection has every vector under the old model; new collection has every vector under the new model. Any step that violates this invariant is a step that breaks rollback.

What rollback actually means

Rollback is not a redeploy; it is a feature-flag toggle. The flag flips back, reads return to the old collection, and because the old collection has been kept complete by dual writes the entire time, no data is lost and no user query goes blank. The cost of rollback is the cost of a flag flip plus a brief cache invalidation, measured in seconds.

If dual writes ever stop while reads are on the new collection, the old collection drifts. Rollback at that point would lose every write that happened in the meantime. That is why step 6 (stop dual writes) comes after the soak, not after the flip.

What backfill before dual-write loses

If the team does step 3 (backfill) before step 2 (dual writes), then every document written during the backfill window only lands in the old collection. The backfill finishes, looks complete, and the new collection silently has a gap shaped exactly like the duration of the backfill. Users get hits on documents that were ingested before the backfill started and after dual writes turned on, but documents ingested during the backfill simply do not exist in retrieval. The bug is hard to detect because aggregate quality metrics look fine; only queries that touch the gap window go wrong.

Step 1 and 2: provision and dual write

Step 3 and 4: backfill and shadow comparison

Step 5 and 6: flag flip, soak, and retirement

Cutover patterns: dual-index, shadow read, and the 2026 default

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Pinecone documents the dual-collection migration pattern explicitly: create a second index, dual write, backfill, flip, retire.
Notion AI's switch from one embedding model family to another used a multi-week dual-write window so any single query could be served by either index during the cutover.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you keep dual writes idempotent across retries when the embedding API rate-limits?

Treat each document's embedding as a versioned operation keyed by (doc_id, model_version). Writes are upserts on that key; retries replay safely. The retry queue persists across process restarts so a transient API outage cannot drop a write silently.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Starting backfill before dual writes are live, which loses every document written during the backfill window because writes only land in the old collection.

60 second bullets to scan on the way to the call.

Why dual writes must precede backfill, not follow it
Why the new collection needs its own index parameters even at the same dimension

Primary sources. Browse if you want the original framing.