Is it true that four data epochs roughly match unique tokens?
Up to about four passes over the same corpus performs close to fully unique tokens. Beyond that knee, each extra pass adds rapidly shrinking value and rising memorization risk.
Think of studying from a small stack of good books while everyone else has a giant library. Reading the stack once teaches you a lot. Reading it a second, third, and fourth time still helps: you catch things you missed and the ideas sink in deeper. But somewhere around the fourth read, something changes. You stop learning new ideas and start memorizing the exact wording on each page. By the tenth read you can recite whole sentences, yet you understand nothing you did not already know. Language models behave the same way with training text. A few repeat passes are nearly as good as fresh material. After that, the repeats turn into empty recitation.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A few years ago the limiting resource in pretraining was compute. You bought as many accelerators as the budget allowed, and the web supplied more text than you could ever consume. That world is gone. Filtering pipelines now discard most of the raw crawl, and dedup removes a large slice of what survives. The pool of high quality unique tokens grows far slower than cluster sizes do.
So a practical question lands on every planning document. The compute budget covers 8T tokens and the corpus holds 2T unique ones. Is training four times over the same data a legitimate move or a quiet way to ruin the run?
Why modern runs are data constrained
A simple expression approximates the compute that a dense transformer run uses:
where N is parameter count and D is training tokens. Chinchilla style analysis says that for a fixed budget C, quality is best when N and D grow together. The best ratio is in the neighborhood of twenty tokens per parameter. A 100B parameter model therefore needs on the order of 2T tokens to be compute optimal. Modern practice deliberately overtrains far past that ratio because inference economics reward smaller models trained longer.
The filtered, deduplicated, high quality slice of the public web is not 100T tokens. Estimates of the usable public text put it within reach of frontier token budgets in the next few years. That is why you hear talk of a data wall. Licensing, transcription, code, multilingual expansion, and synthetic generation all add supply, but each lever has cost, quality, or governance limits.
The result is that a planner regularly faces this situation: compute sized for 8T tokens, corpus holding 2T unique. Something has to give, and the cheapest candidate is repetition. The question is what repetition costs. Answering that needs measurement rather than folklore.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- The data-constrained scaling study behind this number trained over 400 models up to 9B parameters and found up to 4 epochs nearly matches unique data.
- Llama 3 was pretrained on about 15T tokens with aggressive multi level dedup. Meta leaned on data quality and mixing rather than heavy repetition.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy do repeated tokens lose value at all if loss keeps decreasing?
Think about what the model extracts on each pass: general patterns first, then increasingly sequence specific detail that does not transfer.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating repetition as either free or fatal. The measured curve has a knee near four passes: real value up to that point, discounted value after it, and little value past roughly sixteen.
60 second bullets to scan on the way to the call.
What does one epoch mean in a pretraining run?
How many passes over the same data stay close to unique token performance?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.