Chunking strategies: fixed vs semantic vs recursive
Three ways to slice documents before they hit the vector store
Fixed-size chunking is simple; semantic chunks at meaning boundaries; recursive respects the document's own structure. Recursive is the modern default for anything hierarchical (headings, code).
Fixed-size
Glossary →Cut every N tokens (often 256 to 1024) with a small overlap. Fast to implement, format-agnostic, and predictable. Ignores meaning and structure.
Best for: Prototyping and format-agnostic pipelines.
Semantic
Glossary →Split at topic or sentence-similarity boundaries. Adjacent sentences with a big embedding drop become a chunk break. Chunks stay topically coherent; length varies.
Best for: Long-form prose where topics drift naturally.
Recursive
Glossary →Split by the document's own hierarchy: sections, then paragraphs, then sentences, only descending when a chunk is too large. Preserves headings, code fences, and markdown structure.
Best for: Structured docs, markdown, code, technical writing.
At a glance
| Dimension | Fixed-size | Semantic | Recursive |
|---|---|---|---|
| How chunks form | Every N tokens | Embedding-similarity gaps | By hierarchy, then N tokens |
| Boundary respect | None | Topic changes | Section / paragraph / sentence |
| Embedding quality | Medium (mid-topic chunks hurt) | High (coherent chunks) | High (structure preserved) |
| Compute cost to chunk | Cheapest | Extra embedding pass | Document parse plus split |
| Common size | 512 tokens with 50-token overlap | Variable (150 to 900) | Bounded (256 to 1024) |
| Best for | Prototyping, mixed formats | Long-form prose | Markdown, code, structured docs |
Key differences
- 1Fixed-size ignores structure; semantic follows meaning; recursive follows the document's own layout
- 2Fixed-size is trivial to run at scale; semantic needs an extra embedding pass; recursive needs a document parser
- 3Recursive keeps headings with their body text; fixed-size cuts through them; semantic is somewhere in between
- 4Semantic chunks vary in length; fixed and recursive can be bounded
- 5Recursive is the modern default in LangChain / LlamaIndex for markdown, code, and structured docs
In the interview
- Claiming bigger chunks always help recall
- Ignoring that mid-topic breaks hurt embedding quality
- Recommending fixed-size for markdown or code
- Missing that semantic chunk lengths vary and must be bounded
How to choose
Prototype → fixed-size. Long prose → semantic. Structured docs → recursive.
Common misconceptions
Myth: Bigger chunks improve recall.
Reality: Embedding quality degrades past around 512 tokens because the vector averages multiple topics. Recall recovers when chunks stay coherent, not when they grow.
Myth: Recursive is just fixed-size with a fancier splitter.
Reality: It respects the document's headings, code fences, and paragraph boundaries, so chunks correspond to real semantic units instead of arbitrary spans.
Memory aid
Fixed = cut every N pages. Semantic = cut when the topic changes. Recursive = cut respecting the book's chapters and sections.
Can you combine them?
In production stacks these are pipeline stages, not either-or. You can recursively split by structure and then apply a fixed-size cap to bound cost, or run semantic splitting inside a recursive step for very long sections.
Related topics
Related comparisons
Bi-encoder vs Cross-encoder
Two ways embeddings power retrieval and reranking
Hallucination Mitigation vs Context Window Management
Two critical challenges in production LLM systems
Long context vs RAG
Show the model everything, or choose what it should see
RAG vs Fine-tuning
Two approaches to giving LLMs domain specific knowledge
Reranker vs Retriever
The two stages of production RAG, and why you need both
Semantic Search vs Keyword Search
Dense vector similarity vs sparse term matching for retrieval