Find the flawed claim in this dedup policy note.
Click any words you think contain an error. Click again to unmark.
Embedding similarity measures shared topic, not redundancy. Merging every pair above one threshold across domains deletes distinct examples and cuts diversity. Tune per domain and audit removals.
Imagine a librarian told to clear shelf space. Throwing out the twenty photocopies of the same pamphlet is a clear win: nobody needs twenty. Now the librarian gets a new rule: throw out any book whose one-line summary sounds like another book's summary. Suddenly the Italian cookbook goes because a French cookbook is "about cooking". The Spanish edition of a novel goes because the English one tells the same story. In the math section, where every summary says "algebra exercises", almost the whole shelf goes. The library is now smaller and less varied, not more. Embedding dedup with one loose threshold is that second rule. The embedding is a summary of what a text is about, and two texts can share a topic while teaching different things.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Deduplication is one of the most useful cleaning steps in a pretraining pipeline. Exact and near-exact copies waste compute, get memorized, and can leak benchmark items into training. Semantic dedup goes one step further: it embeds each document and removes documents whose embeddings sit very close together.
The policy note pushes this to an extreme: merge every pair above 0.80 cosine similarity, across all domains, because closeness supposedly always improves diversity and robustness. Both the policy and the reason are wrong.
What dedup is for
Web crawls are full of repeats: mirrored pages, boilerplate, license headers, spam templates, and the same article syndicated to many sites. Lee et al. found a single 61-word English sentence repeated more than 60,000 times in C4. Repeats like this cost compute on text the model has already seen, and they push the model to memorize. Models trained on their deduplicated data emitted memorized text about ten times less often. They also found that train-test overlap affected over 4% of the validation sets of standard datasets.
Pipelines usually remove repeats in layers:
- Exact dedup hashes whole documents or long substrings and drops identical ones.
- Fuzzy dedup with MinHash compares sets of word n-grams and catches pages that differ by a few words. FineWeb, for example, used 5-grams and 112 hash functions to find pairs that are about 75% similar.
- Semantic dedup compares embeddings, so it can catch paraphrases and templated pages whose words differ but whose content is the same.
Each layer catches redundancy the one before it misses. Each layer also has more ways to remove text that is not redundant.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Lee et al. found a 61-word English sentence repeated more than 60,000 times in C4. Models trained on deduplicated data emitted memorized text about ten times less often.
- SemDeDup removed 50% of a LAION subset with minimal performance loss, using k-means clusters of embeddings and removing near neighbors inside each cluster.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you pick a semantic dedup threshold for a new domain such as legal text?
Sweep a few thresholds, read sampled pairs near each cut line, and train small proxy models on each version. Compare their downstream scores, not only the removal rate.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming high embedding similarity means two documents are redundant. It often means only that they share a topic, so one global threshold across domains deletes distinct, useful examples.
60 second bullets to scan on the way to the call.
What exact, MinHash, and semantic dedup each detect
What cosine similarity between document embeddings measures
Primary sources. Browse if you want the original framing.
- Lee et al., Deduplicating Training Data Makes Language Models Better
- Abbas et al., SemDeDup: Data-efficient learning at web-scale through semantic deduplication
- Tirumala et al., D4: Improving LLM Pretraining via Document De-Duplication and Diversification
- Penedo et al., The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- Muennighoff et al., Scaling Data-Constrained Language Models
Same topic, related formats. Practice these next.