Chunk size is one of the most important knobs in any RAG pipeline, and the failure shapes tell you which direction to tune.
**Too-large chunks** bring extra material along for the ride. The right paragraph is in there, but so are a few unrelated paragraphs that happened to share the same page or section. The signal to noise ratio per chunk drops. The model has to ignore more to find the answer, and the context block bloats unnecessarily.
**Too-small chunks** split coherent facts. A fact that spans three sentences across the chunk boundary gets cut in half. The retrieval system finds one half (the one that matched the query) and misses the other half. The model receives partial evidence.
Walk through the options:
- **0. Missing specific facts that appear together in the source.** This is the too-small signature. The fact spans more than one chunk and only part of it gets retrieved.
- **1. Right topic plus unrelated paragraph.** This is the too-large signature. Dilution, relevant content arrives bundled with irrelevant content.
- **2. Near-duplicate chunks in top-k.** This is an embedding redundancy or no-MMR problem, not a chunk-size problem. The chunks themselves can be the right size; the retrieval just keeps fetching very similar ones.
- **3. Reranker cannot tell two chunks apart.** This is a reranker capacity issue. A stronger reranker (Cohere Rerank v3, Voyage Rerank-2) handles this; chunk size is unrelated.
The correct answer is option 1.
**The dilution signature is the giveaway.** Each retrieved chunk being on-topic but accompanied by unrelated material is what oversized chunks produce.
### Why oversized chunks dilute
A RAG pipeline fetches the top-k chunks by embedding similarity to the query. The embedding model's view of a chunk is a single vector representing the chunk's average meaning. When the chunk is 1,200 tokens long and contains three different topics, the embedding represents the weighted average of those topics. A query about topic A retrieves the chunk because topic A is in there, but the chunk also delivers topics B and C as collateral.
The model now has to do extra work: read the whole chunk, find the relevant paragraph, ignore the irrelevant ones. Two production consequences:
- **Context bloat**: the top-k now consumes more tokens per chunk, so fewer chunks fit in the budget. You traded depth for width and lost the depth.
- **Lost-in-the-middle**: oversized chunks push relevant content into the middle of the context, where attention is weakest.
### Why each other option is a different failure
**Option 0, missing co-located facts.** This is the too-small signature. A fact that spans multiple sentences ('The XYZ standard, defined in section 4.3, requires a minimum of 8 bytes...') gets cut in the middle by aggressive chunking. The retriever finds the chunk with 'XYZ standard, section 4.3' but misses the chunk with 'minimum of 8 bytes.' Fix: enlarge chunks or add semantic boundary respecting chunking.
**Option 2, near-duplicate chunks in top-k.** This is an embedding redundancy problem. Either the corpus has many similar documents (e.g., versioned docs, FAQ duplicates), or the retrieval lacks diversity (no MMR, no max marginal relevance). Fix: enable MMR with a lambda around 0.5-0.7; deduplicate the corpus at index time.
**Option 3, reranker cannot separate two chunks.** This is a reranker capacity issue.
Quick reference
TL;DR
Right topic plus unrelated paragraph is the dilution signature of oversized chunks. Missing co-located facts is the opposite failure.
Imagine you ask a librarian for a book chapter on cats. Three things can go wrong. First, the librarian hands you a single page about cats but the next sentence is about somebody's grandmother, because the page got cut in the middle of the cat chapter. That is chunks too small. Second, the librarian hands you the cat chapter plus the dog chapter plus a recipe for soup, because all three were stapled together. The cat content is in there but you have to read past a lot of noise to find it. That is chunks too large. Third, the librarian gives you five copies of the same page. That is a problem with the librarian's index, not with the page size. The question asks for the second one.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Very short documents (tweets, log lines) where chunking is essentially the whole document Chunk size loses meaning when the document itself is a single chunk; tune embedding model and rerank instead
Code retrieval where structural boundaries (functions, classes) dominate Token-count chunking is wrong for code; AST-aware chunking is the right tool
Tabular data where rows or cells are the unit Chunking by token count breaks tables; chunk by row or by semantic table section
Quick scan · 2–4 min
Everything important, quickly.
3 minutes: identify option 1 as dilution, name option 0 as the opposite failure (too small), and explain why options 2 and 3 are diversity and reranker problems respectively rather than chunk-size.
Side-by-side
Symptom in retrieved chunks
Likely cause
Fix
On-topic but with unrelated material attached
Chunks too large
Reduce chunk size; semantic-boundary chunking
Co-located facts missing across chunks
Chunks too small
Increase chunk size; add overlap
Near-duplicate chunks in top-k
Diversity gap; corpus redundancy
Enable MMR; deduplicate at index time
Reranker cannot separate two chunks
Reranker capacity
Upgrade reranker (Cohere v3, Voyage 2)
Answer absent from retrieved chunks entirely
Retrieval/embedding mismatch
Try hybrid search; better embeddings
Real-world examples
Real products, models, and research that use this idea.
1LlamaIndex's SemanticSplitterNodeParser chunks at semantic boundaries rather than fixed token counts, reducing the too-small failure mode in 2026 RAG stacks.
2LangChain's RecursiveCharacterTextSplitter is the default chunker for most LangChain-based agents; teams typically tune chunk size 300-500 with 50-100 overlap.
3Cohere Rerank v3 and Voyage Rerank-2 are the dominant 2026 reranker choices for fixing option-3 failure modes; both are commercial APIs with strong cross-encoder backbones.
4BGE-Reranker-v2-Gemma is the self-hosted reranker standard for stacks running on Llama 4 or Qwen 3.5 inference infrastructure.
5Pinecone, Weaviate, and Qdrant all expose MMR as a retrieval parameter; enabling it with lambda 0.5-0.7 is the default fix for option-2 near-duplicate failures.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does chunk overlap interact with rerank?
Sign in or go Pro to see the approach for this follow-up.
QWhy does the right chunk size vary by document type?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the relationship between chunk size and embedding model context length?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Picking option 0 (missing co-located facts) because it 'sounds like a retrieval problem.' It is the opposite failure mode, chunks too small to span the fact. Read the diagnostic shape, not the vibe.