Judge the claim that four data epochs roughly match unique tokens
Up to about four passes over the same corpus performs close to fully unique tokens; beyond that knee, each extra pass adds rapidly shrinking value and rising memorization risk.
Think of studying from a small stack of really good books while everyone else has a giant library. Reading the stack once teaches you a lot. Reading it a second, third, and fourth time still helps: you catch things you missed and the ideas sink in deeper. But somewhere around the fourth read, something changes. You stop learning new ideas and start memorizing the exact wording on each page. By the tenth read you can recite whole sentences, yet you understand nothing you did not already know. Language models behave the same way with training text: a few repeat passes are nearly as good as fresh material, and after that the repeats turn into empty recitation.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A few years ago the limiting resource in pretraining was compute. You bought as many accelerators as the budget allowed, and the web supplied more text than you could ever consume. That world is gone. Filtering pipelines now discard most of the raw crawl, dedup removes a large slice of what survives, and the pool of genuinely high quality unique tokens grows far slower than cluster sizes do.
So a very practical question lands on every planning document: if the compute budget wants 8T tokens and the corpus holds 2T unique ones, is training four times over the same data a legitimate move or a quiet way to ruin the run? This question has an unusually clean empirical answer, and walking through it teaches you how modern labs actually think about data budgets.
Why every modern run is data constrained
Start with the arithmetic that creates the problem. The compute consumed by a dense transformer run is approximated by a simple expression:
where N is parameter count and D is training tokens. Chinchilla style analysis says that for a fixed budget C, quality is best when N and D grow together, with a ratio in the neighborhood of twenty tokens per parameter. A 100B parameter model therefore wants on the order of 2T tokens just to be compute optimal, and modern practice deliberately overtrains far past that ratio because inference economics reward smaller models trained longer.
Now look at supply. The filtered, deduplicated, high quality slice of the public web is not 100T tokens. Estimates of the usable English web sit low enough that frontier token appetites already exceed them, which is why you hear talk of a data wall. Licensing, transcription, code, multilingual expansion, and synthetic generation all add supply, but each lever has cost, quality, or governance limits.
The result is that a planner regularly faces exactly the situation in this question: compute sized for 8T tokens, corpus holding 2T unique. Something has to give, and the cheapest candidate is repetition. The question is what repetition actually costs, and answering that requires measurement rather than folklore.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- The data-constrained scaling study behind this number trained over 400 models up to 9B parameters and found up to 4 epochs nearly matches unique data.
- Llama 3 was pretrained on about 15T tokens with aggressive multi level dedup; Meta leaned on data quality and mixing rather than heavy repetition.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy do repeated tokens lose value at all if loss keeps decreasing?
Think about what the model extracts on each pass: general patterns first, then increasingly sequence specific detail that does not transfer.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating repetition as either free or fatal. The measured curve has a knee near four passes: real value up to that point, steeply discounted value after it, and near zero value past roughly sixteen.
60 second bullets to scan on the way to the call.
What does one epoch mean in a pretraining run?
How many passes over the same data stay close to unique token performance?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.