Which strategy bundle best addresses token scarcity?
Best token-scarcity strategies improve data quality, widen reliable sources, and use synthetic augmentation under strict governance.
Imagine a library running out of good books. Buying piles of duplicate low-quality pamphlets does not help students learn. A better plan is to remove repeated junk, add trusted books from more languages and domains, and use custom practice sheets only with careful checks. A student can also reread a good book a few times and still learn from it. After many rereads, though, the student learns little new. Pretraining token scarcity works the same way: quality and diversity matter more than raw volume.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Token scarcity is the gap between how many tokens a training plan wants and how many unique, high-quality tokens exist. The plan side grows fast. At about 20 tokens per parameter, a compute-optimal 70B model wants 1.4T tokens, and many recent models train well past that ratio. The supply side is fixed: one 2024 estimate projects that training sets will reach the size of the whole stock of public human-written text between 2026 and 2032.
The useful measure is effective tokens, the ones that still teach the model something new. Duplicates, boilerplate and spam count toward raw volume but add little. So the levers that help are the ones that raise effective tokens: better curation, wider trusted sources, and synthetic data used under controls. That is options A, B and C. Options D and E both try to grow one number, parameters or raw volume, and leave the shortage of useful signal in place.
Raw tokens versus effective tokens
A token's value depends on what the model has already seen. The first time the model reads a good paragraph, the loss on it is high and the gradient teaches something. The tenth time, the model mostly predicts it already, so the step spends compute and teaches little. A near-duplicate copy of a page acts the same way as a repeat.
The data-constrained scaling work measured this directly. Training on repeated data for up to about 4 epochs gave almost the same loss as training on the same number of unique tokens. Past that, each extra repeat was worth less. Repeats still gave meaningful gains up to around 16 epochs, and beyond that the returns fell off very fast. So a small amount of repetition is cheap, and heavy repetition is close to wasted compute.
Repetition also has a second cost. Models trained on data with many near-duplicates copy training text more often. In one study, over 1% of unprompted output from such models was copied verbatim from the training set. Deduplication cut that rate by about ten times and reached the same accuracy in fewer steps.
This gives a simple test for any scarcity strategy: does it raise the number of tokens the model has not effectively seen yet? Curation raises it by removing tokens that repeat or carry no signal. New sources raise it by adding text unlike the existing pool. Synthetic data can raise it when it covers real gaps. Growing the model or the raw crawl does not change it much.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- FineWeb filtered and deduplicated 96 Common Crawl snapshots down to about 15T tokens. Its FineWeb-Edu subset, about 1.3T tokens of pages a classifier rated as educational, did better on MMLU and ARC than the larger set.
- The Llama 3 report gives its final pretraining mix as roughly 50% general knowledge, 25% math and reasoning, 17% code and 8% multilingual tokens. That is a deliberate domain rebalance rather than one crawl source in its raw proportions.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you quantify marginal utility of a new corpus source?
Use novelty-adjusted gain on held-out benchmarks and compare against additional token and compute cost.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
A frequent mistake is treating token scarcity as a pure volume problem and ignoring quality, dedup, and governance controls.
60 second bullets to scan on the way to the call.
Token scarcity as information-density issue
Role of dedup and quality filtering
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.