Five near identical copies of one document dominate top-k and starve every other source: diagnose and fix it

Short answer·Medium·4.0 · 0·~3 min·Asked atQualcomm Spotify Writer

Attempt it

In production, your top-k results are repeatedly filled with five near duplicate copies of the same document (a doc that was re-uploaded, mirrored, or lightly edited several times). Other relevant evidence gets pushed out of the context window. Explain why this happens, why it hurts answer quality, and how you would fix it at both ingestion and query time.

Free · 2 AI evals / day

The root of the problem is that retrieval ranks by similarity to the query, and five near-identical copies are all roughly equally similar. Each copy was chunked and embedded separately, so the index holds five chunks that all say nearly the same thing. When a relevant query comes in, all five score high and cluster at the top. The cost is wasted context. Your top-k and the LLM's context window are a fixed budget. Spending five slots on one repeated fact means five different pieces of evidence never reach the prompt. **Recall of the information you actually need drops**, and the answer gets thinner or more one-sided, even though the retriever technically did its job. There are two places to fix this. At **ingestion**, deduplicate before indexing. Exact copies are easy to catch with a content hash. Near-duplicates — lightly edited or reformatted versions — need fuzzy matching, like clustering by embedding similarity or MinHash, so you store one canonical copy. At **query time**, diversify the ranking. Maximal Marginal Relevance (MMR) picks results that are relevant but also different from what it already selected, so the top-k spreads across distinct facets instead of returning paraphrases of one passage.

This is a classic redundancy failure, and the diagnosis comes straight from how retrieval works. Each of the five near-duplicate copies was chunked and embedded independently, so the index contains five distinct vectors sitting very close together in embedding space. Because the ranker scores by similarity to the query, and all five are roughly equally similar, they land in the top-k as a cluster. The damage is to evidence diversity under a fixed budget. The context window holds a limited number of chunks, and burning five slots on the same fact pushes genuinely different chunks out. The retriever returns high-similarity results, but similarity is not usefulness. The LLM sees one perspective repeated rather than the breadth it needs, which lowers effective recall and can bias the answer. The fix works at two layers. **Ingestion-time deduplication** is the durable solution. Catch exact duplicates with content hashing and near-duplicates with embedding clustering or MinHash over shingles, then store a single canonical copy. This removes the redundancy at the source so it can never crowd a result set again, and it saves storage and embedding cost. **Query-time diversification** is the runtime safety net. Maximal Marginal Relevance reranks to balance relevance against novelty: each pick is penalized for being too similar to already-selected results, so the final top-k covers distinct facets. MMR catches duplicates that slipped past ingestion and handles separate documents that overlap. The senior framing is that dedup is the cure and MMR is the insurance — you want both.

TL;DR

Near-duplicate chunks all score alike, so they crowd the top-k and waste the context budget on one repeated fact. Fix it by deduplicating at ingestion and diversifying the ranking with MMR at query time.

Memory aid

Easy to grasp

Imagine you ask five friends for restaurant tips, but four of them just copied the same blog post. You get the same recommendation four times and learn almost nothing new — and you've used up your patience before hearing the one friend with a fresh idea. That's what happens when a document gets uploaded five times: each copy looks just as relevant to the search, so the system hands back five copies of the same passage and shoves the genuinely different information off the page. The fix has two parts. First, before storing anything, notice the copies and keep just one — like deleting duplicate photos on your phone. Second, when picking what to show, reward answers that add something new instead of repeating what you already grabbed.

Key concepts

Retrieval-Augmented Generation Vector Databases Embeddings

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Duplicate flooding is one of those production RAG failures that never shows up in a demo and always shows up at scale. In a clean test corpus every document is unique, retrieval looks great, and the diversity problem is invisible. Then real data arrives — re-uploaded files, mirrored pages, lightly edited revisions of the same policy — and suddenly the top-k is five copies of one thing and the answers get worse for no obvious reason.

What makes this question a good interview probe is that the symptom looks like a ranking failure but is not one. The retriever is doing precisely what it was built to do: return the chunks most similar to the query. The problem is that similarity alone is the wrong objective when the corpus contains redundancy, because it has no concept of how much new information each additional result adds.

This deep dive walks through why near-duplicates cluster in embedding space, reframes the cost in terms of an information budget rather than a recall count, then builds the two-layer fix — ingestion-time deduplication as the durable cure and query-time diversification as the runtime safety net — and explains why a serious system wants both rather than either alone.

Why near-duplicates cluster at the top

Retrieval embeds the query and returns the chunks whose vectors are closest to it. The key fact is that near-duplicate documents produce near-duplicate embeddings: if two passages say almost the same thing, the embedding model maps them to almost the same point in vector space. That is the model working correctly — semantically similar text should land close together.

Now imagine one document was uploaded five times, perhaps with minor edits or reformatting. Each copy was chunked and embedded separately, so the index holds five distinct vectors sitting in a tight cluster. When a query relevant to that content arrives, the query vector is close to all five, so all five score high.

A top-k ranker by cosine similarity has no tiebreaker that says "I already have this fact." It simply takes the k highest scores, and if the five highest all belong to the duplicate cluster, that is what you get. Raising k does not help — it just pulls in more of the cluster before reaching anything new.

The deeper issue is that relevance ranking optimizes the wrong quantity. It maximizes per-chunk similarity to the query, when what you actually want is to maximize the total distinct information delivered to the model under a fixed budget. Those two objectives diverge exactly when the corpus contains redundancy.

Reframing the cost: it is an information budget, not a recall count

Layer one: deduplication at ingestion

Layer two: diversity at query time with MMR

Why a serious system runs both layers

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

LangChain and LlamaIndex both ship an MMR retriever mode precisely to break up near-duplicate clusters in the top-k
Web-scale corpora are routinely deduplicated with MinHash/LSH before indexing because mirrored and re-hosted pages are everywhere

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the lambda parameter in MMR change behavior, and how would you tune it?

MMR scores each candidate as lambda·relevance minus (1−lambda)·max similarity to selected. Lambda near 1 is almost pure relevance (duplicates return); lambda near 0 maximizes diversity (may surface off-topic results). Tune it on a labeled set, watching whether answers improve with more distinct evidence without losing the genuinely best chunk.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Reaching only for query-time MMR and leaving the duplicates in the index — the redundancy keeps wasting embedding storage and can still surface, so dedup at ingestion is the durable fix MMR only backstops.

60 second bullets to scan on the way to the call.

Why near-duplicate chunks all score similarly and cluster in the top-k
Why this wastes the context budget and lowers effective recall

Primary sources. Browse if you want the original framing.