The data wall is where unique, high-quality human text runs short of what scaling needs, so growth must come from repetition, wider sources or synthetic data, and quality control becomes the limit.
Think of a gold mine. At first, every shovel of earth holds gold, and the miners dig as fast as they can. Over time the rich veins run out. There is still plenty of rock, but each ton holds less gold, and digging deeper costs more. Training text works the same way. The web holds a huge amount of text, but much of it is junk or copies. The good, original text is limited, and the biggest training runs are close to using all of it. After that point, teams can re-read the same good text a few more times, look for new kinds of text, or have models write new text. Each option helps, and each has a cost.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Scaling laws say that bigger models need more training tokens. The data wall is the point where the supply of new, high-quality human text can no longer keep up. There is still plenty of raw text on the web. But the clean, original part is limited, and the largest training runs are close to using it.
The consequence is a change in what limits progress. Below the wall, teams mostly decide how much compute to spend. Near it, they must decide how to repeat, widen and generate data, and how to keep quality high while doing so.
Demand: how many tokens scaling asks for
Compute-optimal training uses about 20 tokens per parameter:
By this rule, a 70B model needs about 1.4T tokens, a 405B model about 8.1T, and a 1T-parameter model about 20T.
Real runs often go well past this. A smaller model trained for longer is cheaper to serve, so teams overtrain: they spend extra training compute to save inference compute later. Llama 3 8B trained on about 15T tokens. That is about 1,900 tokens per parameter, nearly 94 times its compute-optimal count. Llama 3 70B trained on about the same 15T tokens, about 10 times its count. GPT-3 175B, by contrast, trained on about 300B tokens in 2020.
So demand grows from two sides. Frontier models get bigger, which raises the compute-optimal count. And overtraining multiplies the count for models of every size, because inference cost matters more as more people use a model. Training sets grew from about 300B tokens to about 15T tokens, a factor of 50, in about four years.
Every one of those tokens should ideally be new. A token that repeats earlier text teaches less than a fresh one, and the gap grows with each repeat. That is what turns token demand into a data problem.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Epoch AI estimates about 300T tokens of usable public human text (90% range 100T to 1,000T) and projects that frontier training sets reach it between 2026 and 2032.
- Llama 3 8B trained on about 15T tokens, close to 1,900 tokens per parameter, far past the compute-optimal count of about 20.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide how many epochs to allow for each data domain?
Count unique deduplicated tokens per domain, compare them with the planned share, and keep valuable domains near or under 4 epochs unless ablations show more repetition still helps.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking the data wall means the internet is out of text. Raw text is plentiful. What runs short is unique, high-quality text. The limit depends on how strictly a team filters and how much it repeats.
60 second bullets to scan on the way to the call.
Definition of the data wall
Compute-optimal token demand and overtraining
Primary sources. Browse if you want the original framing.
- Villalobos et al., Will we run out of data? Limits of LLM scaling based on human-generated data
- Muennighoff et al., Scaling Data-Constrained Language Models
- Hoffmann et al., Training Compute-Optimal Large Language Models
- Shumailov et al., AI models collapse when trained on recursively generated data
- Gunasekar et al., Textbooks Are All You Need
Same topic, related formats. Practice these next.