How can you detect redundancy across retrieved chunks before they hit the prompt?

Flashcard·Medium·4.0 · 0·~30s·Asked atCohere Databricks OpenAI

Attempt it

Cheap method: exact or near exact text hash (normalized casing, whitespace, punctuation). Catches copy-paste duplicates across docs. Slightly heavier method: pairwise cosine similarity between chunk embeddings with a threshold (for example, drop the lower-ranked chunk when similarity is above 0.92). At assembly time, run the cheap hash first because it is constant per chunk and catches the most common duplicates, then run pairwise cosine only over what survives. The full O(k squared) cosine pass is only over the small top-k after dedupe and rerank, so it stays cheap.

Your answer

Retrieved chunks that say the same thing are still distractors. They burn token budget, push the actually-distinct chunks toward the lost-in-the-middle trough, and dilute the model's attention. Detecting redundancy before context assembly matters as much as detecting irrelevance. There are two practical detection methods, and they catch different things. **Text-hash dedupe (cheap).** Normalize each chunk, lowercase, strip whitespace, remove punctuation, and hash it. Two chunks with the same hash are textual duplicates. This catches the common case of the same content appearing in multiple documents (FAQ pages, syndicated articles, version-controlled docs). Cost is constant per chunk and the comparison is O(k) using a set. **Embedding-cosine dedupe (heavier).** Compute cosine similarity between each pair of chunk embeddings. If the similarity exceeds a threshold (commonly 0.90-0.95), the lower-ranked chunk is dropped. This catches paraphrased duplicates, same meaning, different words, that the hash misses. Cost is O(k squared) in similarity comparisons. At assembly time the right shape is to run hash dedupe first on the wide retrieved set (top-30 or top-50), then run cosine dedupe only on the trimmed reranked set (top-5 or top-10). The expensive pass operates on a small k, so the squared cost stays manageable.

**Two layers of dedupe handle the two flavors of duplication you actually see in retrieval.** The first layer is **text-hash dedupe**. Normalize each chunk (lowercase, collapse whitespace, strip punctuation) and hash it; two chunks with the same hash are byte-identical after normalization. This catches the dominant case: the same content syndicated across multiple documents, copy-pasted FAQs, version-controlled documentation, near-duplicate scrapes. Cost is one hash per chunk and a set membership check, so it scales linearly with the retrieved set size. The second layer is **embedding-cosine dedupe**. Compute cosine similarity between each pair of chunk embeddings in the candidate set; drop the lower-ranked chunk whenever similarity is above a threshold (commonly 0.90-0.95, tuned per corpus). This catches paraphrased duplicates that the hash misses: the same fact stated in different words, two chunks that summarize the same passage at different granularities. The cost is O(k squared) similarity comparisons. The assembly-time pipeline layers them. Hash dedupe runs on the full retrieval set (top-30 or top-50). The hash is constant per chunk and catches the obvious duplicates cheaply. Then rerank runs on what survives. Then cosine dedupe runs on the small reranked set (top-5 or top-10). The expensive pass is only over a small k, so the squared cost is trivial. The one to highlight at assembly time is **embedding-cosine dedupe** because it is the layer that catches semantic redundancy after rerank, which is the redundancy that actually reaches the model's prompt.

Quick reference

TL;DR

Normalized text hash catches verbatim duplicates cheaply; pairwise cosine similarity over embeddings catches paraphrased duplicates; layer them so the expensive pass only sees the small set.

Memory aid

Easy to grasp

Imagine sorting a stack of leaflets a campaign just brought back. Some leaflets are obvious photocopies of each other, same words, same layout. You can flip through fast and toss those by eye in seconds. Others are reworded versions of the same message; you actually have to read them to notice. So you do the fast visual pass first across the whole stack and only do the careful reading on the much smaller pile that survives. Retrieval dedupe works the same way. A quick visual scan for the photocopies, a careful read for the reworded ones, in that order.

Key concepts

Context engineering Embeddings Retrieval-augmented generation

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 minutes: the two duplicate flavors, hash dedupe shape and cost, cosine dedupe shape and cost, layered ordering, threshold tuning, edge cases.

Real-world examples

Real products, models, and research that use this idea.

LlamaIndex ships a SimilarityPostprocessor node that implements cosine dedupe at the top-k stage of its query pipeline.
LangChain's EmbeddingsRedundantFilter operates the same way and pairs with the contextual-compression retriever.
Cohere's 2026 production RAG guide explicitly recommends layered dedupe (hash before rerank, cosine after rerank).
Anthropic's Claude Opus 4.7 RAG cookbook flags near-duplicate retrieval as a top source of irrelevance-penalty regressions.
Vespa, Weaviate, and Qdrant ship native dedupe operators in 2026 that let the work happen at the index layer before chunks ever reach the prompt builder.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you tune the cosine threshold for a new corpus?

QWhat happens when two near-duplicates disagree on one important fact?

QCan dedupe happen at the index level instead of at retrieval time?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Running pairwise cosine similarity over the whole top-50 retrieval set. The cost scales with k squared. The right shape is hash-first over the wide set, then cosine over the small reranked set.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.