Design an idempotent embedding indexing pipeline that survives partial failures

Short answer·Hard·4.0 · 0·~3 min·Asked atCoinbase Salesforce Turbopuffer

Attempt it

You're building the indexing pipeline that ingests documents, chunks them, embeds the chunks, and upserts to a vector DB. The pipeline must be idempotent: re-running the same input must not produce duplicates, must handle partial failure retries cleanly, and must support model upgrades without leaking stale vectors. Outline the design.

Free · 2 AI evals / day

The whole design rests on one principle: **content-keyed identifiers at every stage**. If a chunk's id is derived from its content, then rerunning the pipeline produces the same id. Which means the vector DB sees an upsert to an existing row rather than a fresh insert. Retries become no-ops automatically. **Chunking.** Use a deterministic chunker (fixed-size, semantic boundary, or hierarchical) and compute `chunk_id = sha256(doc_id + chunk_position + chunker_version)`. Same input, same chunker, same chunk_ids every time. **Embedding.** Cache embeddings by `(sha256(chunk_text), model_id, model_version)`. A re-embed under the same model is a cache hit. A new model is a new cache namespace, so the upgrade does not touch the old embeddings. **Upsert.** The vector DB row key is `(chunk_id, model_version)`. Writing the same row twice is harmless because upsert semantics are by key. During a model migration, the same chunk_id can hold vectors under multiple model_versions in parallel. State tracking. A per-doc state table records the pipeline stage each doc has reached (chunked, embedded, indexed). On retry, the pipeline picks up at the right stage instead of redoing finished work. Failure handling. Transient errors retry with exponential backoff. Permanent errors (oversized docs, invalid encoding) go to a dead-letter queue for manual triage. Garbage collection. A periodic job deletes vectors for chunk_ids that no longer exist in the source corpus, and old model_versions after the migration safety window closes.

The pipeline must be **idempotent under retries** and non-destructive under model upgrades. Both reduce to the same primitive: content-keyed identifiers at every stage. **Chunking.** Deterministic chunker with a versioned implementation. `chunk_id = sha256(doc_id || chunk_position || chunker_version)`. Same input, same chunk_ids forever. Bumping `chunker_version` creates a new namespace for migrations. **Embedding.** Cache by `(sha256(chunk_text), model_id, model_version)`. Re-embedding under the same model is a cache hit; a new model is a new namespace. **Upsert.** Vector DB primary key is `(chunk_id, model_version)`. Upsert-by-key makes duplicate writes no-ops; multiple model_versions coexist during migration. Pipeline driver. At-least-once queue (Kafka, SQS, DB-backed). Per-doc state table tracks stages (chunked, embedded, indexed) so partial reruns resume cleanly. Failure handling. Transient errors retry with exponential backoff. Permanent failures move to a dead-letter queue tagged with doc_id and stage. Model upgrade. Re-embed at `(chunk_id, new_model_version)` while old vectors stay live. Validate recall@k, then atomically flip query traffic by changing the queried model_version. Rollback is symmetric. Garbage collection. Periodic jobs remove orphaned chunk_ids and old model_versions after the safety window closes. GC is cheap because every row is keyed deterministically.

TL;DR

Content-keyed IDs at every stage (sha256-derived chunk_id, model_version-namespaced upserts) make retries no-ops and model upgrades non-destructive.

Memory aid

Easy to grasp

Picture a librarian copying books onto shelves. If she labels each copy with a random sticker, she cannot tell which copies are duplicates. If a book is dropped halfway and she restarts, she might end up with two copies of the same chapter on different shelves. The fix is to label every chapter with a tag derived from the book title and the chapter number. Now if she restarts, she just writes the same chapter into the same labeled slot. Nothing duplicates because the slot already exists. When the librarian gets a new translation of the same books, she does not throw out the old translation. She builds a parallel shelf labeled with the new translation, lets readers pick which version they want, and only removes the old shelf once everyone has switched. Indexing pipelines work the same way.

Key concepts

Embeddings Vector Databases

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

An indexing pipeline ingests documents, chunks them, embeds the chunks, and upserts vectors into a vector database. It runs at scale, on noisy inputs, with multiple stages that can fail independently, and it has to stay correct under three pressures: retries (the queue redelivers a work item), partial failures (the embed step succeeds but the upsert step does not), and model upgrades (the embedding model version changes, but the old vectors still need to serve traffic until the new ones are validated).

The design that handles all three reduces to a single primitive: content-keyed identifiers used as natural primary keys at every stage. If the chunk id is derived from the document id plus position plus chunker version, then rerunning the chunker on the same document produces the same chunk ids. If the embedding cache is keyed by chunk text plus model version, then re-embedding under the same model is a hit and a new model is a new namespace. If the vector store's primary key is (chunk_id, model_version), then upserts are no-ops on duplicate writes and the old and new model versions coexist during a migration.

The rest of the design (state tables, dead-letter queues, garbage collection) supports observability and lifecycle management around that core primitive.

Why content-keyed IDs are the core primitive

A naive pipeline assigns random UUIDs to chunks and vectors. The first time you retry a failed batch, the queue redelivers the same documents, the pipeline assigns fresh UUIDs to the same chunks, and the vector database accumulates duplicates. You now have two copies of every retried chunk, no automated way to tell which is which, and a recall problem in production because retrieval surfaces both copies.

The fix is to derive every identifier from the input content. chunk_id = sha256(doc_id || chunk_position || chunker_version) is deterministic: same inputs produce the same chunk_id forever. The vector DB row key becomes (chunk_id, model_version). When the pipeline reruns, it writes to the same row, and upsert by key semantics turn the duplicate write into a no-op.

This primitive solves the retry problem cleanly. It also solves the model-upgrade problem, because the model_version dimension in the row key lets the same chunk_id have multiple vectors under different model versions. The old vectors stay live while you write the new ones; once you validate the new model, you flip the query layer to read the new version atomically. Rollback is symmetric.

The chunker_version in the chunk_id deserves its own mention. Chunking strategies change too: sliding-window 512 tokens becomes semantic chunking on heading boundaries becomes hierarchical chunking with parent docs. Without versioning the chunker, a chunker change would silently overwrite the old chunks, and you would lose the ability to A/B test the chunker choice. Versioning the chunker creates a new chunk_id namespace and lets both strategies live in the index simultaneously.

The embedding cache and why model_version belongs in the key

The state table and per-doc observability

Failure handling and the dead-letter queue

Model upgrades as parallel write then flip

Garbage collection

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

import hashlib

CHUNKER_VERSION = "semantic-v3"
MODEL_ID = "voyage-3-large"
MODEL_VERSION = "2026-04"

def chunk_id(doc_id: str, chunk_position: int) -> str:
    raw = f"{doc_id}|{chunk_position}|{CHUNKER_VERSION}"
    return hashlib.sha256(raw.encode()).hexdigest()

def cache_key(chunk_text: str) -> str:
    text_hash = hashlib.sha256(chunk_text.encode()).hexdigest()
    return f"{text_hash}|{MODEL_ID}|{MODEL_VERSION}"

def upsert_vector(vector_db, doc_id, chunk_position, vector, chunk_text):
    cid = chunk_id(doc_id, chunk_position)
    # Composite key is the idempotency primitive
    vector_db.upsert(
        id=f"{cid}|{MODEL_VERSION}",
        vector=vector,
        metadata={
            "doc_id": doc_id,
            "chunk_id": cid,
            "chunk_position": chunk_position,
            "chunker_version": CHUNKER_VERSION,
            "model_id": MODEL_ID,
            "model_version": MODEL_VERSION,
        },
    )

Real products, models, and research that use this idea.

Pinecone's reference architecture for production RAG includes a `model_version` namespace in metadata and recommends content-keyed chunk IDs for safe rollback.
Weaviate's multi-tenant pattern uses one tenant per (corpus, model_version), which gives the same isolation as composite-key upserts and makes migration a tenant-level operation.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you handle a chunker change that produces a different chunk count for the same document?

Bump chunker_version. The old chunks live under the old version, the new chunks live under the new version, both indexed in parallel. Validate retrieval quality against a labeled set under the new chunker. Once green, flip the query layer to read the new chunker_version and GC the old chunks after a safety window. No row gets overwritten; the migration is parallel write then flip.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Using random UUIDs as chunk_ids. Retry now writes a duplicate vector under a new id, and the DB has no way to tell the two are the same chunk.

60 second bullets to scan on the way to the call.

Define content-keyed chunk_id from doc_id, position, and chunker_version.
Explain why model_version belongs in the cache key and vector primary key.

Primary sources. Browse if you want the original framing.