You are tuning chunk size in a RAG pipeline. What is the core tradeoff between very small chunks (e.g. 128 tokens) and very large chunks (e.g. 2000 tokens)?

Chunking is how you break a long document into pieces small enough to embed and store in a vector index. The size of each piece matters because it controls two opposite things at once. When chunks are tiny, say 128 tokens, each one captures a tight semantic unit, often a single sentence. The retriever can land on the exact line that answers the query. The catch is that the LLM only sees that one line and may not have enough surrounding context to write a confident answer. When chunks are huge, say 2000 tokens, each one carries a whole section of context. The LLM has plenty to work with. But the single vector that represents the chunk has to average together many different ideas, which makes the embedding blurrier and harder to match against a specific query. Most teams land in the middle around 256 to 512 tokens, then layer a trick called parent-document retrieval where you embed small chunks but return their bigger parent paragraph to the model.

Chunk size is one of the highest leverage knobs in a RAG pipeline. It controls two opposing pressures at once. **Small chunks** (around 128 to 256 tokens) embed a tight semantic unit, often a sentence or two. The query pins precisely to the exact fact that answers it. Top-k recall on specific facts is high. The cost: the model receives a fragment with little surrounding context. Answers come out under-grounded or stitched together. **Large chunks** (1500 to 2000 plus tokens) give the model rich, self-contained passages, which helps faithfulness and reasoning. But the embedding model must compress that whole passage into one vector. That single vector is a blurry average across multiple ideas, so queries for a specific fact land near many irrelevant chunks and recall drops. The production answer is rarely a single number. Pick a middle size, 256 to 512 tokens with sentence boundary splitting, then use parent-document retrieval: embed small chunks for precision, return the larger parent paragraph for context. Measure with retrieval recall@k and end-to-end faithfulness, not chunk size in isolation.