Flashcard: what does chunking mean in RAG, and why do we chunk at all?

**Chunking** is the step in a RAG pipeline where you take a long document and split it into smaller pieces before you turn anything into vectors. Each piece is called a chunk, and a typical chunk is 200 to 800 tokens, roughly a few paragraphs. There are three reasons we chunk instead of embedding whole documents. First, retrieval precision. A single embedding for a 50-page PDF is a blurry summary of every topic in that PDF. When a user asks about one paragraph, the document's vector might not match closely, and even if it does, you would not know which paragraph is relevant. Smaller chunks have sharper meanings, so the retriever can find the right paragraph. Second, context-budget economics. Even with frontier models like Claude Opus 4.7 or Gemini 3.1 that allow huge inputs, every token costs latency and money. You only want to send the relevant pieces, not the whole document. Third, citation granularity. Chunks are the natural unit for source attribution. The answer can point at chunk 3 of doc X, which is a clickable citation.

**Chunking** is the indexing-time step that splits long source documents into smaller passages, typically 200 to 800 tokens, before each passage is embedded and stored. It is the most consequential RAG decision after choice of embedding model, and getting it wrong is the most common cause of low retrieval quality. Three motivations make chunking non-optional. The first is retrieval precision. A document level embedding averages every topic the document covers; the resulting vector lives in a vague centroid that matches many queries weakly and none of them strongly. Chunk-level embeddings carve the document into pieces, each with a sharper semantic signature, so the retriever can surface the one paragraph that actually answers the question. The second is context-window economics. Even with million token windows on Claude Opus 4.7, Gemini 3.1, and GPT-5.5, every token costs latency and dollars. And 'lost-in-the-middle' attention degradation means more tokens does not always mean better answers. You want to ship only the relevant chunks. The third is citation granularity. Chunks are the unit of attribution in the rendered answer, so a chunk size that matches a typical claim is what makes citations useful. Chunking happens once, at indexing time. It is not a query-time operation, and reindexing is the cost of changing chunking strategy.