Flashcard: what does chunking mean in RAG, and why do we chunk at all?
Chunking splits long documents into 200 to 800 token pieces before embedding so retrieval is precise, prompts stay cheap, and citations land on specific paragraphs.
Imagine you are studying for an open book test and someone hands you a 50-page textbook. If you only know how to grab whole books from the shelf, every search drops the entire book on your desk. Useful pages and useless pages all together. Chunking is the opposite habit: before the test, you tear the book into single-page handouts, label each one, and shelve them separately. Now when you ask a question, you can fetch only the two or three pages you actually need. The book did not change; the way you sliced it did. In RAG, every document gets sliced into small pieces called chunks before anything is stored away. The retriever then finds the right chunk, not the right book. That precision is why chunking is one of the most consequential decisions in a RAG system.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Chunking is the RAG decision that sounds boring and turns out to control most of your retrieval quality. The slogan version (split the doc into pieces and embed each piece) hides every interesting choice: how big, by what boundary, with how much overlap, with how much surrounding context, in what order, with which strategy per content type. Teams that treat chunking as a configuration value tend to also treat their retrieval problems as embedding problems, and then spend weeks tuning the wrong knob.
The right way to think about chunking is that it determines two things at once. It determines the unit of retrieval (what the index actually searches over) and the unit of attribution (what a citation can point at). Those two units want to be the same thing, so the chunk has to be small enough to be specific and large enough to be self-contained.
This deep dive defines chunking precisely, walks through the three motivations that make it non-optional, names the strategies that work for different content types, and explains why getting chunking right is more leverage than getting embeddings right.
What chunking is, and why it is indexing-time only
Chunking is the step where you take a long document and split it into smaller passages, called chunks, before any embedding happens. A typical chunk in 2026 production systems is 200 to 800 tokens, with some teams running smaller (100 to 250) for FAQ-style content and some running larger (up to 1500) for long-form documents where reasoning across paragraphs matters.
The step runs once per document, at the moment that document enters the index. Each chunk is embedded, the vector goes into the vector database alongside the chunk text and a chunk identifier, and the original document is usually retained as a parent reference for follow-up lookups. Nothing about chunking happens at query time. The query is embedded as one piece; the retriever returns whole chunks; the prompt assembler concatenates them.
The operational consequence is that chunking decisions are sticky. If you change chunk size, overlap, or boundary strategy, you must re-embed and re-index the entire corpus. For a million chunk corpus, that is a meaningful compute and dollar cost. This is why senior teams evaluate chunking against a measured retrieval-quality benchmark before they scale the corpus, not after.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangChain's RecursiveCharacterTextSplitter splitting Markdown documents along headers then paragraphs, the default in many production RAG stacks.
- LlamaIndex's SemanticSplitterNodeParser using embedding similarity to merge sentences into coherent chunks instead of fixed-size windows.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you choose a chunk size for a mixed corpus of Markdown docs, PDFs, and source code?
Different splitters per content type; Markdown by heading, PDFs by paragraph, code by syntactic unit; quote a chunk-size band per modality.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Embedding entire documents as single vectors. The vector becomes a blurry average of every topic the doc covers, retrieval loses precision, and the prompt has to carry the whole document just to answer one question.
60 second bullets to scan on the way to the call.
Why a whole document embedding loses retrieval precision
Typical chunk-size band for prose and why it sits there
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.