Why deduplicate retrieved chunks before stuffing them into context?

A vector store returns the top-k chunks most similar to the query. Similarity to the query is its only criterion. If five chunks happen to all be near-identical (because the same paragraph was copy-pasted across five docs, or because a FAQ was reproduced in five places), all five will score about the same against the query and all five will show up in the top-k. The cost shows up in two places. First, you are paying for the same tokens five times, each duplicate chunk consumes its full slice of the context budget. Second, and more important, those five slots could have held five different facts. Every duplicate is a slot that is no longer available for new information. Deduping at the assembly stage fixes this. The pipeline checks the candidates against each other before writing them into the context block. Exact duplicates can be caught with a simple hash on the chunk text. Near-duplicates need an embedding similarity check between candidates, if two chunks have cosine similarity above some threshold like 0.95, treat them as the same chunk and keep only one. The result is denser context. The same five slots now carry five distinct facts. The model sees more of what the knowledge base actually contains.

Top-k similarity retrieval optimizes one thing: similarity between each candidate chunk and the query. It has no penalty for similarity between candidates. That blind spot is what makes dedupe necessary at the assembly stage. The failure case is concrete. Suppose a knowledge base has a one-paragraph definition of a term reproduced verbatim in five different documents, a common situation for product specs, license text, or company-wide policy paragraphs. A query that targets that definition will pull all five copies into the top-5 because they are all maximally similar to the query and roughly equally similar to each other. The context block ends up as five repeats of the same paragraph. The immediate cost is token spend. Five copies at 200 tokens each is 1,000 tokens to communicate one 200-token fact. The deeper cost is opportunity. Those four duplicate slots could have held different chunks, a related caveat, an exception, a worked example, a neighbouring concept. Dedupe converts wasted slots into new information. Mechanically, dedupe runs after retrieval and before assembly. A cheap exact-text hash catches verbatim copies. A more thorough near-duplicate pass uses pairwise embedding cosine between candidates and collapses any pair above a threshold like 0.92-0.95. The surviving chunks are then ordered for assembly (often via MMR or a reranker) and written into the context.