How can you detect redundancy across retrieved chunks before they hit the prompt?

Flashcard·Medium·4.0 · 0·~30s·Asked atDataiku Replicate

Attempt it

Cheap method: exact or near exact text hash (normalized casing, whitespace, punctuation). Catches copy-paste duplicates across docs. Slightly heavier method: pairwise cosine similarity between chunk embeddings with a threshold (for example, drop the lower-ranked chunk when similarity is above 0.92). At assembly time, run the cheap hash first because it is constant per chunk and catches the most common duplicates, then run pairwise cosine only over what survives. The full O(k squared) cosine pass is only over the small top-k after dedupe and rerank, so it stays cheap.

Your answer

Retrieved chunks that say the same thing are still distractors. They burn token budget, push the actually-distinct chunks toward the lost-in-the-middle trough, and dilute the model's attention. Detecting redundancy before context assembly matters as much as detecting irrelevance. There are two practical detection methods, and they catch different things. **Text-hash dedupe (cheap).** Normalize each chunk, lowercase, strip whitespace, remove punctuation, and hash it. Two chunks with the same hash are textual duplicates. This catches the common case of the same content appearing in multiple documents (FAQ pages, syndicated articles, version-controlled docs). Cost is constant per chunk and the comparison is O(k) using a set. **Embedding-cosine dedupe (heavier).** Compute cosine similarity between each pair of chunk embeddings. If the similarity exceeds a threshold (commonly 0.90-0.95), the lower-ranked chunk is dropped. This catches paraphrased duplicates, same meaning, different words, that the hash misses. Cost is O(k squared) in similarity comparisons. At assembly time the right shape is to run hash dedupe first on the wide retrieved set (top-30 or top-50), then run cosine dedupe only on the trimmed reranked set (top-5 or top-10). The expensive pass operates on a small k, so the squared cost stays manageable.

**Two layers of dedupe handle the two flavors of duplication you actually see in retrieval.** The first layer is **text-hash dedupe**. Normalize each chunk (lowercase, collapse whitespace, strip punctuation) and hash it; two chunks with the same hash are byte-identical after normalization. This catches the dominant case: the same content syndicated across multiple documents, copy-pasted FAQs, version-controlled documentation, near-duplicate scrapes. Cost is one hash per chunk and a set membership check, so it scales linearly with the retrieved set size. The second layer is **embedding-cosine dedupe**. Compute cosine similarity between each pair of chunk embeddings in the candidate set; drop the lower-ranked chunk whenever similarity is above a threshold (commonly 0.90-0.95, tuned per corpus). This catches paraphrased duplicates that the hash misses: the same fact stated in different words, two chunks that summarize the same passage at different granularities. The cost is O(k squared) similarity comparisons. The assembly-time pipeline layers them. Hash dedupe runs on the full retrieval set (top-30 or top-50). The hash is constant per chunk and catches the obvious duplicates cheaply. Then rerank runs on what survives. Then cosine dedupe runs on the small reranked set (top-5 or top-10). The expensive pass is only over a small k, so the squared cost is trivial. The one to highlight at assembly time is **embedding-cosine dedupe** because it is the layer that catches semantic redundancy after rerank, which is the redundancy that actually reaches the model's prompt.

TL;DR

Normalized text hash catches verbatim duplicates cheaply; pairwise cosine similarity over embeddings catches paraphrased duplicates; layer them so the expensive pass only sees the small set.

Memory aid

Easy to grasp

Imagine sorting a stack of leaflets a campaign just brought back. Some leaflets are obvious photocopies of each other, same words, same layout. You can flip through fast and toss those by eye in seconds. Others are reworded versions of the same message; you actually have to read them to notice. So you do the fast visual pass first across the whole stack and only do the careful reading on the much smaller pile that survives. Retrieval dedupe works the same way. A quick visual scan for the photocopies, a careful read for the reworded ones, in that order.

Key concepts

Context engineering Embeddings Retrieval-augmented generation

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Redundancy in retrieved chunks is one of the under-discussed sources of context-engineering damage. Duplicate chunks burn token budget, displace genuinely useful content, and dilute the model's attention exactly the way unrelated distractors do. Detecting them before assembly is cheap when done right and expensive when done wrong.

This card walks through the two practical detection methods (hash and cosine), where each one belongs in the pipeline, how to tune them, and the edge cases that catch teams who only run one layer.

Two flavors of duplicate

Real corpora produce two distinct shapes of duplication, and the detection method has to match.

Verbatim duplicates. The same chunk text appearing in two different document IDs. Causes: syndicated content (the same article on multiple sites), version-controlled documentation (v2 and v3 of the same doc), copy-pasted FAQs, repeated boilerplate. After light normalization (lowercase, collapse whitespace, strip punctuation), these chunks hash to the same value.

Paraphrased duplicates. Two chunks that say the same thing in different words. Causes: different writers covering the same fact, the same source summarized at different granularities, translation back and forth in multilingual corpora. These chunks do not match by hash. They cluster in embedding space.

Why both layers matter

Hash alone misses paraphrases. Cosine alone is too expensive to run over wide retrieval sets and can be too aggressive on technical corpora where slight wording differences encode important distinctions. The layered approach catches both flavors at the right cost.

Hash dedupe in detail

Embedding-cosine dedupe in detail

The layered pipeline

Edge cases and tuning over time

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

LlamaIndex ships a SimilarityPostprocessor node that implements cosine dedupe at the top-k stage of its query pipeline.
LangChain's EmbeddingsRedundantFilter operates the same way and pairs with the contextual-compression retriever.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you tune the cosine threshold for a new corpus?

Curate a labeled set of duplicate and non-duplicate pairs; sweep threshold from 0.85 to 0.97; pick the value with best F1 on duplicate detection without dropping known-distinct pairs.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Running pairwise cosine similarity over the whole top-50 retrieval set. The cost scales with k squared. The right shape is hash-first over the wide set, then cosine over the small reranked set.

60 second bullets to scan on the way to the call.

How do you distinguish verbatim duplicates from paraphrased duplicates?
What does hash-based detection catch and what does it miss?

Primary sources. Browse if you want the original framing.