Design an idempotent embedding-indexing pipeline that survives partial failures

Short answer·Hard·4.0 · 0·~3 min

Attempt it

You're building the indexing pipeline that ingests documents, chunks them, embeds the chunks, and upserts to a vector DB. The pipeline must be idempotent: re-running the same input must not produce duplicates, must handle partial-failure retries cleanly, and must support model upgrades without leaking stale vectors. Outline the design.

Free · 2 AI evals / day

The whole design rests on one principle: **content-keyed identifiers at every stage**. If a chunk's id is derived from its content, then rerunning the pipeline produces the same id. Which means the vector DB sees an upsert to an existing row rather than a fresh insert. Retries become no-ops automatically. **Chunking.** Use a deterministic chunker (fixed-size, semantic boundary, or hierarchical) and compute `chunk_id = sha256(doc_id + chunk_position + chunker_version)`. Same input, same chunker, same chunk_ids every time. **Embedding.** Cache embeddings by `(sha256(chunk_text), model_id, model_version)`. A re-embed under the same model is a cache hit. A new model is a new cache namespace, so the upgrade does not touch the old embeddings. **Upsert.** The vector DB row key is `(chunk_id, model_version)`. Writing the same row twice is harmless because upsert semantics are by key. During a model migration, the same chunk_id can hold vectors under multiple model_versions in parallel. State tracking. A per-doc state table records the pipeline stage each doc has reached (chunked, embedded, indexed). On retry, the pipeline picks up at the right stage instead of redoing finished work. Failure handling. Transient errors retry with exponential backoff. Permanent errors (oversized docs, invalid encoding) go to a dead-letter queue for manual triage. Garbage collection. A periodic job deletes vectors for chunk_ids that no longer exist in the source corpus, and old model_versions after the migration safety window closes.

The pipeline must be **idempotent under retries** and non-destructive under model upgrades. Both reduce to the same primitive: content-keyed identifiers at every stage. **Chunking.** Deterministic chunker with a versioned implementation. `chunk_id = sha256(doc_id || chunk_position || chunker_version)`. Same input, same chunk_ids forever. Bumping `chunker_version` creates a new namespace for migrations. **Embedding.** Cache by `(sha256(chunk_text), model_id, model_version)`. Re-embedding under the same model is a cache hit; a new model is a new namespace. **Upsert.** Vector DB primary key is `(chunk_id, model_version)`. Upsert-by-key makes duplicate writes no-ops; multiple model_versions coexist during migration. Pipeline driver. At-least-once queue (Kafka, SQS, DB-backed). Per-doc state table tracks stages (chunked, embedded, indexed) so partial reruns resume cleanly. Failure handling. Transient errors retry with exponential backoff. Permanent failures move to a dead-letter queue tagged with doc_id and stage. Model upgrade. Re-embed at `(chunk_id, new_model_version)` while old vectors stay live. Validate recall@k, then atomically flip query traffic by changing the queried model_version. Rollback is symmetric. Garbage collection. Periodic jobs remove orphaned chunk_ids and old model_versions after the safety window closes. GC is cheap because every row is keyed deterministically.

Quick reference

TL;DR

Content-keyed IDs at every stage (sha256-derived chunk_id, model_version-namespaced upserts) make retries no-ops and model upgrades non-destructive.

Memory aid

Easy to grasp

Picture a librarian copying books onto shelves. If she labels each copy with a random sticker, she cannot tell which copies are duplicates. If a book is dropped halfway and she restarts, she might end up with two copies of the same chapter on different shelves. The fix is to label every chapter with a tag derived from the book title and the chapter number. Now if she restarts, she just writes the same chapter into the same labeled slot. Nothing duplicates because the slot already exists. When the librarian gets a new translation of the same books, she does not throw out the old translation. She builds a parallel shelf labeled with the new translation, lets readers pick which version they want, and only removes the old shelf once everyone has switched. Indexing pipelines work the same way.

Key concepts

Embeddings Vector Databases

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

60s: content-keyed chunk_id at chunking stage; embedding cache keyed by (chunk_hash, model_id, model_version); vector primary key (chunk_id, model_version) for non-destructive upgrades; per-doc state table; dead-letter queue for permanent failures; GC for orphaned chunks and old model_versions.

Code example

python

import hashlib

CHUNKER_VERSION = "semantic-v3"
MODEL_ID = "voyage-3-large"
MODEL_VERSION = "2026-04"

def chunk_id(doc_id: str, chunk_position: int) -> str:
    raw = f"{doc_id}|{chunk_position}|{CHUNKER_VERSION}"
    return hashlib.sha256(raw.encode()).hexdigest()

def cache_key(chunk_text: str) -> str:
    text_hash = hashlib.sha256(chunk_text.encode()).hexdigest()
    return f"{text_hash}|{MODEL_ID}|{MODEL_VERSION}"

def upsert_vector(vector_db, doc_id, chunk_position, vector, chunk_text):
    cid = chunk_id(doc_id, chunk_position)
    # Composite key is the idempotency primitive
    vector_db.upsert(
        id=f"{cid}|{MODEL_VERSION}",
        vector=vector,
        metadata={
            "doc_id": doc_id,
            "chunk_id": cid,
            "chunk_position": chunk_position,
            "chunker_version": CHUNKER_VERSION,
            "model_id": MODEL_ID,
            "model_version": MODEL_VERSION,
        },
    )

Real-world examples

Real products, models, and research that use this idea.

Pinecone's reference architecture for production RAG includes a `model_version` namespace in metadata and recommends content-keyed chunk IDs for safe rollback.
Weaviate's multi-tenant pattern uses one tenant per (corpus, model_version), which gives the same isolation as composite-key upserts and makes migration a tenant-level operation.
Anthropic's contextual retrieval guide explicitly describes content-keyed chunk IDs and per-version cache namespaces as the recommended pattern for safe re-indexing.
Qdrant's collection per model version pattern is the documented 2026 best practice for embedding pipelines that need rollback semantics across model upgrades.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you handle a chunker change that produces a different chunk count for the same document?

QWhat's the failure mode if you forget model_version in the upsert key?

QHow does the state-tracking table interact with at least once queues?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Using random UUIDs as chunk_ids. Retry now writes a duplicate vector under a new id, and the DB has no way to tell the two are the same chunk.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.