Walk through orchestrating a 500M-vector re-embedding backfill: rate limits, checkpoints, verification, and cutover readiness.

A backfill of this size is a small distributed system, not a script. The work has four jobs: load fast enough, load safely, verify, and cut over. **Throughput.** 500M docs over 14 days is about 413 per second sustained. You provision the embedding API quota with headroom (600 to 800 per second) and you throttle ingest into the vector database so you do not damage live query p99. Off-peak windows help. **Resumability.** Partition the corpus into ordered ranges by stable primary key. Each range is a unit of work, processed independently and checkpointed in a durable store (Postgres, S3 manifest, whatever). When a worker crashes, the next worker picks up unfinished ranges. Every write is an upsert keyed by doc ID plus embedding model version, so retries are safe. **Racing with dual writes.** Dual writes are already writing fresh vectors to the new collection. The backfill must not overwrite a fresher dual-write vector with a stale historical one. Version-check before writing, or skip docs the dual-write path already touched. **Verification and cutover.** Count parity per range, spot-check sampled vectors, and run the recall harness in the new embedding space. Cut over only when all gates pass and a rollback flag is tested.

Five workstreams in parallel. - **Throughput math.** 500M in 14 days is ~413 docs/sec sustained. Provision embedding API quota at 600 to 800 docs/sec with retry budget, confirm the vector database bulk-ingest path absorbs that rate without breaching live query p99, and throttle dynamically based on live latency. - **Partition and checkpoint.** Split the corpus into ranges by stable primary key (e.g. 5,000 ranges of 100K docs each). A control table records per-range status: pending, in-flight, done, failed. Workers claim ranges via atomic claim or lease, process them, and update status on completion. Restart-safe. - **Idempotent upserts.** Write key is (doc_id, model_version). Retrying a range never duplicates. Critically, the dual-write path uses the same key, so the two writers naturally converge if you also enforce a write-wins-by-timestamp rule: backfill writes only if no fresher dual-write exists for that doc. - **Cost controls.** Estimate API spend up front (tokens per doc x price x 500M). Hash content and skip unchanged documents. Burn-rate alerts catch a runaway worker. Track spend per range. - **Verification and cutover.** Count parity per range. Re-embed sampled docs and compare vectors. Run the recall eval harness against the new collection with ground truth in the new embedding space. Run shadow traffic for a soak period and compare recall plus latency. Cut over with a flag, keep dual writes on for a soak, and have a tested rollback.