Hoffmann et al. (Chinchilla, 2022) argued GPT-3 was severely under-trained. Walk through what they actually showed and how it changed architecture / training choices over the following two years.
Free · 2 AI evals / day
**Chinchilla showed GPT-3 was too big and trained too little.** Hoffmann et al. (2022) ran a careful sweep of model sizes and training token counts at fixed compute budgets and found a clear pattern: for any compute budget, the loss-optimal allocation is roughly 20 tokens per parameter.
GPT-3 had 175 billion parameters trained on 300 billion tokens, a ratio of ~1.7 tokens-per-param. Chinchilla itself was 70 billion parameters trained on 1.4 trillion tokens, a ratio of 20 tokens-per-param, at the SAME training compute as GPT-3. The 70B Chinchilla beat the 175B GPT-3 on essentially every benchmark.
**What changed after 2022.** Labs stopped pushing parameter counts upward and started pushing training tokens upward. Llama-1 7B (2023) trained on 1T tokens. Llama-2 7B (2023) trained on 2T tokens. Llama-3 8B (2024) trained on 15T tokens. The architectures stayed nearly identical (pre-norm, RMSNorm, SwiGLU, GQA); the training budget grew almost 15x in two years.
**The twist:** by 2024 the field was deliberately training FAR PAST Chinchilla-optimal. Llama 3.1 8B at 15T tokens has a ratio of ~1875 tokens-per-param, ninety times Chinchilla's optimum. The reason is that inference cost dominates training cost for any model that gets actually deployed. A smaller model trained on much more data costs more to train but is cheaper to serve forever after.
**What Chinchilla actually measured.** Hoffmann et al. (2022) trained over 400 transformer models across a grid of parameter counts (70M to 16B) and training token counts (5B to 500B) and fit a scaling law to the resulting loss surface. Their headline finding: for a fixed total training compute `C`, the loss-optimal allocation between parameters `N` and training tokens `D` is approximately `N ∝ D`, with the ratio `D/N ≈ 20`. In other words, you should train roughly 20 tokens per parameter.
**Comparison with GPT-3.** GPT-3's 175B parameters trained on 300B tokens corresponds to `D/N ≈ 1.7`, an order of magnitude below the Chinchilla optimum. At the same training compute, Chinchilla showed you should instead train a 70B-parameter model on 1.4T tokens (ratio 20). They did the experiment: their 70B Chinchilla model outperformed the 175B Gopher and the 175B GPT-3 across MMLU, BIG-bench, reading comprehension, and reasoning tasks. The conclusion: GPT-3-era models were under-trained, not too small.
**The architectural footprint was tiny.** Chinchilla itself was structurally nearly identical to Gopher: same decoder-only stack, same attention, same FFN, same pre-norm placement. The change was purely in the parameter-token allocation. This is important to articulate: scaling laws are about training budget allocation, not about which architectural ingredient you use.
**How the field absorbed this between 2022 and 2024.** The first wave was correction: Llama-1 (Feb 2023) trained 7B on 1T tokens, ratio ~143, deliberately above Chinchilla because Meta wanted small, cheap to serve models. Llama-2 (July 2023) doubled the 7B token budget to 2T (~285 ratio). Llama-3 (April 2024) jumped to 15T tokens on 8B (~1875 ratio). Mistral 7B (Sept 2023) followed the same pattern.
**Why the field walked past Chinchilla-optimal.** Chinchilla minimizes loss per training FLOP.
Quick reference
TL;DR
Chinchilla (Hoffmann et al., 2022) showed that for a fixed compute budget the optimal mix is about 20 training tokens per parameter, not the ~1.7 tokens-per-param that GPT-3 had.
Imagine you have a fixed amount of flour to bake bread. You can either make one giant loaf or several smaller, more-kneaded ones. For a long time, AI labs were making giant loaves and not kneading them enough. Chinchilla showed that the same flour split into smaller loaves and kneaded much longer makes better bread. So instead of growing models forever, the field started training smaller models on much more data. And then they noticed something else: once you bake the bread, you serve it to customers for years. A smaller loaf is cheaper to slice every single time someone orders one. So even though it costs more flour-time to knead the smaller loaf, you save money every day forever after. That is why a 2026 8-billion-parameter model trained on 15 trillion tokens beats a 2020 175-billion-parameter model trained on 300 billion tokens.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Training a one-off research model that will not be deployed Chinchilla-optimal IS the right allocation when there is no inference cost to amortize. Walking past it wastes training compute.
Frontier-pushing model where the goal is quality at any cost (not deployment economics) If you have the compute and want the best possible loss, the Chinchilla curve at very large scale may still be where you want to operate. Walking past optimum makes sense mainly for deployment-cost reasons.
Domain-specialized training on a small corpus If your target corpus is bounded (medical, low-resource language, niche code), token-starvation makes Chinchilla irrelevant; the binding constraint is data quantity, not compute allocation.
Quick scan · 2–4 min
Everything important, quickly.
6 min: Chinchilla's compute-optimal ratio of 20 tokens-per-param + how it differed from Kaplan + GPT-3 was under-trained + the methodological fix + architectural shape barely changed + post-2022 the field walked PAST optimum + inference cost dominates training cost + Llama-1 to Llama-3 progression + where Chinchilla still applies.
Side-by-side
Model
Year
Params
Tokens
Tokens/Param
Regime
GPT-3
2020
175B
300B
~1.7
Pre-Chinchilla, severely under-trained
Chinchilla
2022
70B
1.4T
20
Compute-optimal
Llama-1 7B
2023
7B
1T
~143
First step past Chinchilla
Llama-2 7B
2023
7B
2T
~285
Walking past Chinchilla
Llama-3 8B
2024
8B
15T
~1875
Far past Chinchilla, inference-optimized
Llama 3.1 405B
2024
405B
15.6T
~39
Past Chinchilla for the frontier dense model
Real-world examples
Real products, models, and research that use this idea.
1Hoffmann et al. (Chinchilla, 2022): 70B model on 1.4T tokens beat 175B GPT-3 and 280B Gopher at the same training compute.
2Llama-1 7B (Feb 2023): 1T tokens, ratio ~143, deliberately above Chinchilla; demonstrated that small models trained heavily can compete with much larger Chinchilla-optimal models.
3Llama-3 8B (Apr 2024): 15T tokens, ratio ~1875, ninety times Chinchilla-optimal; current best small-model benchmark on most tasks.
4Llama 3.1 405B (Jul 2024): 15.6T tokens, ratio ~39, still well past Chinchilla, used as the largest dense Llama model.
5DeepSeek V3 (Dec 2024): 671B total / 37B active MoE, trained on ~14.8T tokens; uses MoE specifically to keep active parameters small for serving while keeping total parameters large for quality.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy do MoE models change the Chinchilla calculus?
Sign in or go Pro to see the approach for this follow-up.
QIf you only have 1e22 training FLOPs, what model size and token count should you pick?
Sign in or go Pro to see the approach for this follow-up.
QHow does data quality interact with the Chinchilla scaling law?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reading Chinchilla as a recipe to FOLLOW. The field followed it for about a year and then deliberately walked past it; modern Llama 3.1 8B trains at ~1875 tokens per param, almost 100x the Chinchilla ratio.