Chunk overlap: what it protects against and what too much of it costs
Overlap keeps a boundary straddling idea inside at least one chunk; too much overlap just duplicates text and floods retrieval with near identical hits.
Imagine tearing a long letter into pages with a straight rip. Sometimes the rip lands in the middle of an important sentence, so neither page makes full sense on its own. Now imagine you instead let the bottom few lines of each page reappear at the top of the next page. The sentence that got cut now lives complete on at least one page, so when you go looking for it later, you find the whole thought. That repeated strip is overlap. But if you copy half of every page onto the next one, your stack of pages doubles, and when you search you keep pulling out the same paragraph three times. So you want a thin shared strip, not a thick one.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Almost every RAG quality problem traces back to retrieval, and most retrieval problems trace back to how the corpus was cut into chunks. Chunking is the unglamorous preprocessing step that decides what the smallest retrievable unit of knowledge actually is. Get it wrong and no reranker, no clever query rewrite, and no bigger model fully recovers, the relevant fact was already mangled before it ever reached the index.
Overlap is one of the two knobs on the simplest chunker, sitting right beside chunk size. It looks trivial, which is exactly why people set it carelessly in both directions. This walkthrough builds up why the knob exists, what each extreme costs, how to pick a value, and when to stop reaching for overlap and fix the splitter instead.
What a chunk boundary really is
A fixed size splitter walks the document and emits a new chunk every N characters or tokens. The boundary between two chunks is therefore a position chosen by counting, not by meaning. It has no idea whether it is landing between two paragraphs, in the middle of a sentence, or right after the word 'not' that flips the meaning of the clause that follows.
This is the root cause of the whole overlap discussion. Natural language packs meaning into spans that cross arbitrary character offsets: a term and its definition, a claim and its qualifier, a pronoun and the noun it refers to. When the blind cut lands inside one of those spans, you get two half-chunks. The first holds the setup with no payoff, the second holds the payoff with no setup.
At retrieval time this hurts twice. The embedding of each half-chunk is computed from incomplete text, so it sits in the wrong place in vector space and matches the query poorly. And even if one half is retrieved, the generator receives a fragment that may be useless or, worse, misleading. The boundary did not just lower recall, it can corrupt the meaning of what you do retrieve.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangChain's RecursiveCharacterTextSplitter exposes chunk_size and chunk_overlap as the two core knobs, defaulting overlap to a small fraction of chunk size.
- LlamaIndex's SentenceSplitter sets a modest default overlap so a sentence near a boundary appears in both neighbouring nodes.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does overlap change the total number of chunks and your embedding cost?
Reason from the multiplier: overlap fraction f scales chunk count by about 1/(1-f), so derive ingestion embedding spend, storage, and per query distance cost from that.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Setting overlap to zero to save space, then losing facts that sit on a chunk boundary; or cranking overlap so high that the index doubles and retrieval returns the same passage repeatedly.
60 second bullets to scan on the way to the call.
State the purpose of chunk overlap in one breath
Explain what a chunk boundary is and why it lands arbitrarily
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.