Keeping a RAG index fresh as the source corpus changes: what does incremental indexing require?
The documents behind a RAG system are constantly created, edited, and deleted. Explain how to keep the index fresh incrementally rather than rebuilding it, and what you must handle for deletions.
Key chunks to a stable document id, detect changes by hash or timestamp, upsert edits as delete then insert, and hard-delete removals so stale content can never be retrieved.
Picture a library where the shelves are your search index and the catalogue cards point readers to the right shelf. Every day some books get added, some get a new edition, and some get pulled. You would not burn down the library and rebuild it from scratch each morning. Instead you give every book a fixed ID, check which ones changed since yesterday, and only touch those. A new edition means you take the old copy off the shelf and put the new one up under the same ID. A pulled book means you actually remove it, not just slip a 'do not lend' note inside while leaving it on the shelf — because if it is still on the shelf, a reader can still grab it. A RAG index works the same way: update only what moved, and make deletions real.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A RAG index is never finished. The moment you ship, the corpus behind it starts drifting: a support article gets rewritten, a policy page is retired, a new product launches with fresh docs. The index has to track that drift, or your system confidently answers from text that no longer exists. Freshness is not optional — a stale retrieval grounds the generator in the wrong facts, and a stale generator is indistinguishable from a hallucinating one to the user.
The naive instinct is to rebuild the index on a schedule. That works until it doesn't: at tens of millions of chunks, a full re-embed and re-index costs hours and a large compute bill, and you still serve stale results between runs. The real discipline is incremental indexing — treating the vector store as a derived view you keep in sync with a primary source. This walkthrough covers the identity you need, how to detect what changed, how to apply edits without leaving orphans, and why deletions deserve special care.
Why a full rebuild does not scale
A full rebuild reads the entire corpus, re-chunks it, embeds every chunk, and writes a fresh index. Each of those steps scales linearly with corpus size, and the embedding step in particular is the expensive one — it is per-chunk model inference, often against a paid API.
At a thousand documents this is fine; you might rebuild nightly and never think about it. At ten million documents producing a hundred million chunks, a rebuild is hours of compute and a four- or five-figure embedding bill every time. Worse, the cadence of a rebuild bounds your freshness: if you rebuild nightly, a document edited at 9am is wrong in the index until tomorrow's run finishes.
The insight is that on any given day only a tiny fraction of the corpus actually changed. Rebuilding spends almost all of its effort re-processing documents that are byte for byte identical to what is already indexed. Incremental indexing flips this: do work proportional to the change, not to the corpus. That is the only model that stays affordable and keeps freshness tight as the corpus grows.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LlamaIndex's IngestionPipeline ships a docstore-backed upsert that hashes each document and skips, updates, or deletes nodes so re-running the pipeline only touches changed files.
- Pinecone and Weaviate expose upsert by id and delete by id so a sync job can replace or remove exactly the chunks for one document without rebuilding the collection.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you keep deletions from leaking when access to a document is revoked rather than the document being removed?
Distinguish content deletion from access change; discuss per-tenant or per-user metadata filters enforced at query time plus removing the vector from any shared index it should no longer appear in.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Soft-deleting a removed document by flagging it instead of deleting its vectors — the chunks stay searchable and get retrieved, which is a correctness bug and, for revoked-access or right to be forgotten content, a compliance violation.
60 second bullets to scan on the way to the call.
Explain why full rebuilds do not scale for a changing corpus
State what identity each chunk must carry to support incremental sync
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.