Which of these are valid considerations when choosing a chunking strategy for RAG?
Chunk size is a tradeoff between semantic coherence, downstream context size, and embedding cost.
Imagine cutting a book into pieces. Cut too small, you lose meaning. Cut too large, the model gets overwhelmed. The sweet spot depends on what you're searching for.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Chunking is the most under-appreciated lever in a RAG pipeline. It determines what units of meaning your retriever can match against, and once you've chosen poorly, no amount of reranking can fully recover the lost signal.
A chunk that splits a sentence in half loses both halves' meaning. A chunk that bundles five unrelated ideas dilutes the embedding so it matches everything weakly and nothing strongly. The art is finding chunks that are semantically coherent enough to embed cleanly but small enough to leave room for diversity in the top-k.
The three axes that matter
Boundary preservation: chunks should respect natural breaks, paragraphs, sections, function definitions. Fixed size splits at 512 tokens are a baseline, not a solution. Embedding-aware semantic splitters look at where embedding distance jumps and break there.
Context budget: chunk_size × top_k must fit your model's context window with room for the system prompt and the answer. If your LLM has 8k context and you reserve 2k for prompt+answer, 6k is your retrieval budget. With chunk_size=512 you can include 12 chunks; with 1024 you only fit 6.
Embedding cost: embedding 100k chunks at 100 tokens each is cheap. Embedding 1M chunks at 50 tokens each is 10× more expensive at index time and at every re-index. This rarely dominates the LLM cost but it shows up at scale.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangChain RecursiveCharacterTextSplitter, splits by paragraph → sentence → word, preserving structure.
- LlamaIndex SentenceWindowNodeParser, index at sentence level but return surrounding context window.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat about chunking code or tables specifically?
AST-based or row-based chunking, never raw character split
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating chunking as a fixed size hyperparameter to tune rather than a design choice tied to content shape.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.