Sequence the offline indexing pipeline that prepares a corpus for retrieval
- 1Clean and normalize the text (strip boilerplate, fix encoding)
- 2Split the cleaned text into retrievable chunks
- 3Upsert the vectors plus metadata into the index
- 4Embed each chunk into a vector with the embedding model
- 5Load and parse the source documents into raw text
The offline index pipeline is load, clean, chunk, embed, upsert — each stage feeds the next, and skipping the clean before chunk order corrupts every downstream vector.
Imagine prepping vegetables before cooking. First you bring the groceries in and unpack them (load and parse). Then you wash off the dirt and trim bad bits (clean and normalize). Only then do you chop everything into bite-size pieces (chunk). Next you weigh and label each piece so you can find it later (embed into a vector). Finally you put the labeled pieces into the right drawers in the fridge (upsert into the index). The order is not optional: if you chop before washing, the dirt ends up inside every piece, and no amount of labeling later fixes that. Indexing a document corpus follows the same discipline — wash the whole thing before you cut it.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Most RAG quality problems are retrieval quality problems, and most retrieval quality problems trace back to the indexing pipeline — the offline work that turns a messy corpus into a searchable index. It is easy to treat this as plumbing and get the order wrong, and the cost of a wrong order is not a crash. It is silent: the system runs, returns results, and quietly retrieves noise.
The pipeline has five stages, and their order is not a style choice. It is dictated by a chain of data dependencies, where each stage can only run once the previous stage has produced what it needs. Understanding those dependencies is the difference between memorizing a sequence and being able to reason about where a new technique slots in or what reruns when a model changes.
This deep dive walks the five stages in order, explains the dependency that forces each transition, dwells on the clean before chunk constraint that trips up most candidates, and covers how the pipeline behaves under incremental updates and model migrations in production.
The five stages and the dependency chain
The pipeline is load, clean, chunk, embed, upsert. Read it as a chain where the output type of each stage is the input type of the next.
Load and parse takes heterogeneous files — PDFs, HTML, Word, Markdown — and produces raw text. Clean and normalize takes raw text and produces cleaned text: boilerplate stripped, encoding fixed, whitespace collapsed. Chunk takes cleaned text and produces a list of retrievable units. Embed takes each chunk and produces a vector. Upsert takes vectors plus metadata and produces entries in the index.
Notice that the order falls out of the types. The embedding model's input is text, and specifically the final text units, so embed must come after chunk. Chunking's input is text you have decided is worth indexing, so it comes after cleaning. And upsert has nothing to write until vectors exist, so it is always last. You could not reorder these even if you wanted to — each stage is blocked until its predecessor has run.
The one stage with real freedom is where metadata extraction lives. Source id, title, date, and section labels are usually captured during load or cleaning and carried forward so they can be attached at upsert, making the index filterable later.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangChain document loaders parse files, then text splitters chunk, then an embedding model and vector store finish the pipeline in that fixed order.
- LlamaIndex separates IngestionPipeline transformations so cleaning and metadata extraction run before the node-splitting and embedding steps.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhere does contextual retrieval fit into this order, and why is it not a reordering?
Explain that contextual retrieval prepends an LLM-generated summary of the surrounding document to each chunk before it is embedded. That is a new step inserted between chunk and embed, so the chunk text the model sees carries document context. The five-stage backbone is unchanged; you are enriching the chunk, not moving a stage.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Chunking before cleaning. Boilerplate, nav menus, and broken encoding then leak into chunk boundaries and poison the embeddings — and no later stage can undo it.
60 second bullets to scan on the way to the call.
List the five stages in order: load, clean, chunk, embed, upsert
Explain why each stage depends on the output of the previous one
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.