Yes. At equal compute, a smaller model trained on more tokens can beat a larger, undertrained one: Chinchilla 70B on 1.4T tokens beat Gopher 280B on 300B tokens.
A coach has a fixed number of practice hours to share across a team. One plan hires a huge squad, so each player gets a few minutes of practice. The other plan hires a smaller squad, so each player gets hours. On paper the huge squad has more talent, but most of its players never learn the plays. The smaller, well-drilled squad wins the match. A language model has the same budget problem. Parameters are the squad and training tokens are the practice. Training compute is roughly parameters times tokens, so more parameters means fewer tokens each. Gopher had 280 billion parameters and saw 300 billion tokens. Chinchilla had 70 billion parameters and saw 1.4 trillion tokens on the same budget, and Chinchilla scored higher.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Parameter count is the most visible number on a model card, so it is easy to read it as a quality score. At a fixed training budget it is not. Training compute is shared between parameters and tokens, and a model that spends too much of its budget on size ends up training on too little data.
The Chinchilla study turned this into a measured result. It found that most large models of 2020 and 2021 were undertrained. A model one quarter the size of Gopher, trained on four times as much data with the same compute, beat Gopher on most tasks.
One budget, two ways to spend it
Training a dense transformer costs close to six floating-point operations per parameter per token. The forward pass takes about two (one multiply and one add for each weight). The backward pass takes about four, because it computes gradients for both the activations and the weights.
C is total training FLOPs, N is the parameter count and D is the number of training tokens. Once C is fixed by the cluster size and the calendar, N and D are no longer independent. Choose N, and D follows as C / 6N.
Gopher's training budget was about 5.76e23 FLOPs. Spent on a 280B model, that budget buys about 343B tokens, roughly one token per parameter. Spent on a 70B model, it buys about 1.37T tokens, roughly 20 per parameter. Both runs cost the same. The open question is which one learns more.
A larger model can learn more from each token, because it has more capacity to store patterns. It also needs to see enough examples to fill that capacity. A 280B model that sees each kind of pattern only a few times cannot use most of its parameters well. That is what undertrained means: training stopped while the loss was still falling quickly, because the budget went to size instead of data.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Chinchilla 70B, trained on 1.4T tokens with Gopher's compute budget, beat Gopher 280B, GPT-3 175B, Jurassic-1 178B and Megatron-Turing NLG 530B on most evaluated tasks.
- LLaMA-13B, trained on 1T tokens, beat GPT-3 175B on most benchmarks while using about a quarter of GPT-3's training compute.
What an interviewer would ask next. Try answering before peeking at the approach.
QAt a fixed budget of 1e24 FLOPs, what model size and token count does the 20 tokens per parameter rule give?
Put D = 20N into C = 6ND, so N = sqrt(C / 120): about 91B parameters trained on about 1.8T tokens.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reading parameter count as a quality score. At fixed compute a bigger model gets fewer tokens, and an undertrained large model can lose to a smaller one.
60 second bullets to scan on the way to the call.
Training compute as a function of parameters and tokens
Why fixed compute forces a size and token trade-off
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.