Post-Chinchilla analyses treat GPT-3 as undertrained because its parameter scale was paired with too few tokens for compute-optimal balance.
Think of a fixed budget for a factory. You can spend most of it on a huge building full of machines and have little left for raw material. Or you can build a smaller plant and buy far more material. The huge plant sits half idle: its machines could make much more, but the material runs out. GPT-3 was the huge plant. It had 175B parameters (the machines) but trained on only 300B tokens (the material), under 2 tokens per parameter. In 2022 the Chinchilla study looked at the same budget. It showed that a plant about a third the size, with more than three times the material, makes a better product. The extra material keeps the smaller model learning for longer.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
GPT-3 was the largest dense language model of 2020: 175B parameters, trained on about 300B tokens. Two years later, DeepMind's Chinchilla study called it undertrained. The word does not mean GPT-3 was broken. It means its compute budget was split badly: too much went into parameters and too little into training tokens. So the same compute could have produced a lower loss.
The diagnosis rests on one equation for training cost, one empirical result about how loss scales, and one direct test at full size.
The budget equation and GPT-3's split
Training a dense transformer costs about 6 FLOPs per parameter per token: about 2 in the forward pass and about 4 in the backward pass.
For GPT-3, N = 175B and D = 300B give 3.15 × 10²³ FLOPs. That matches the 3.14 × 10²³ FLOPs (3,640 petaflop/s-days) reported in the GPT-3 paper. The ratio D/N is about 1.7 tokens per parameter.
The estimate leaves out the attention scores themselves, which grow with context length. Kaplan and colleagues put that extra forward cost at about 2 · n_layer · n_ctx · d_model per token. For GPT-3, with 96 layers, a 2,048-token context, and a width of 12,288, that is about 1.4% of the 2N forward cost. This is small enough to ignore.
With C fixed, choosing N fixes D. A planner can spend the budget on a huge model that sees few tokens, or on a smaller model that sees many. Both ends waste compute. A huge model stopped early is far from what its size could learn, and its loss is still falling steeply when training ends. A tiny model trained for a very long time hits the limit of its capacity, and extra tokens buy almost nothing. The best split sits between the two ends, and Chinchilla set out to find it.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- GPT-3 175B trained on 300B tokens with 3.14 × 10²³ FLOPs; training covered only 0.44 epochs of its 410B-token filtered Common Crawl.
- Chinchilla (70B, 1.4T tokens) used the same compute as Gopher (280B, 300B tokens) and beat it on MMLU, 67.5% against 60.0%.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhere does the 6 in C ≈ 6ND come from?
Count matrix-multiply FLOPs per parameter per token: about 2 in the forward pass and about 4 in the backward pass.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Explaining undertraining with context window or alignment stages instead of token balance.
60 second bullets to scan on the way to the call.
The C ≈ 6ND budget and GPT-3's compute
GPT-3's tokens per parameter
Primary sources. Browse if you want the original framing.
- Brown et al., Language Models are Few-Shot Learners
- Kaplan et al., Scaling Laws for Neural Language Models
- Hoffmann et al., Training Compute-Optimal Large Language Models
- Besiroglu et al., Chinchilla Scaling: A replication attempt
- Porian et al., Resolving Discrepancies in Compute-Optimal Scaling of Language Models
Same topic, related formats. Practice these next.