Chinchilla's lesson: how did it reshape architecture and training choices after 2022?

Short answer·Hard·4.0 · 0·~3 min·Asked atAnthropic OpenAI·Relevant atGoogle

Attempt it

Hoffmann et al. (Chinchilla, 2022) argued GPT-3 was severely under-trained. Walk through what they actually showed and how it changed architecture / training choices over the following two years.

Free · 2 AI evals / day

**Chinchilla showed GPT-3 was too big and trained too little.** Hoffmann et al. (2022) ran a careful sweep of model sizes and training token counts at fixed compute budgets and found a clear pattern: for any compute budget, the loss-optimal allocation is roughly 20 tokens per parameter. GPT-3 had 175 billion parameters trained on 300 billion tokens, a ratio of ~1.7 tokens-per-param. Chinchilla itself was 70 billion parameters trained on 1.4 trillion tokens, a ratio of 20 tokens-per-param, at the SAME training compute as GPT-3. The 70B Chinchilla beat the 175B GPT-3 on essentially every benchmark. **What changed after 2022.** Labs stopped pushing parameter counts upward and started pushing training tokens upward. Llama-1 7B (2023) trained on 1T tokens. Llama-2 7B (2023) trained on 2T tokens. Llama-3 8B (2024) trained on 15T tokens. The architectures stayed nearly identical (pre-norm, RMSNorm, SwiGLU, GQA); the training budget grew almost 15x in two years. **The twist:** by 2024 the field was deliberately training FAR PAST Chinchilla-optimal. Llama 3.1 8B at 15T tokens has a ratio of ~1875 tokens-per-param, ninety times Chinchilla's optimum. The reason is that inference cost dominates training cost for any model that gets actually deployed. A smaller model trained on much more data costs more to train but is cheaper to serve forever after.

**What Chinchilla actually measured.** Hoffmann et al. (2022) trained over 400 transformer models across a grid of parameter counts (70M to 16B) and training token counts (5B to 500B) and fit a scaling law to the resulting loss surface. Their headline finding: for a fixed total training compute `C`, the loss-optimal allocation between parameters `N` and training tokens `D` is approximately `N ∝ D`, with the ratio `D/N ≈ 20`. In other words, you should train roughly 20 tokens per parameter. **Comparison with GPT-3.** GPT-3's 175B parameters trained on 300B tokens corresponds to `D/N ≈ 1.7`, an order of magnitude below the Chinchilla optimum. At the same training compute, Chinchilla showed you should instead train a 70B-parameter model on 1.4T tokens (ratio 20). They did the experiment: their 70B Chinchilla model outperformed the 175B Gopher and the 175B GPT-3 across MMLU, BIG-bench, reading comprehension, and reasoning tasks. The conclusion: GPT-3-era models were under-trained, not too small. **The architectural footprint was tiny.** Chinchilla itself was structurally nearly identical to Gopher: same decoder-only stack, same attention, same FFN, same pre-norm placement. The change was purely in the parameter-token allocation. This is important to articulate: scaling laws are about training budget allocation, not about which architectural ingredient you use. **How the field absorbed this between 2022 and 2024.** The first wave was correction: Llama-1 (Feb 2023) trained 7B on 1T tokens, ratio ~143, deliberately above Chinchilla because Meta wanted small, cheap to serve models. Llama-2 (July 2023) doubled the 7B token budget to 2T (~285 ratio). Llama-3 (April 2024) jumped to 15T tokens on 8B (~1875 ratio). Mistral 7B (Sept 2023) followed the same pattern. **Why the field walked past Chinchilla-optimal.** Chinchilla minimizes loss per training FLOP.

Quick reference

TL;DR

Chinchilla (Hoffmann et al., 2022) showed that for a fixed compute budget the optimal mix is about 20 training tokens per parameter, not the ~1.7 tokens-per-param that GPT-3 had.

Memory aid

Easy to grasp

Imagine you have a fixed amount of flour to bake bread. You can either make one giant loaf or several smaller, more-kneaded ones. For a long time, AI labs were making giant loaves and not kneading them enough. Chinchilla showed that the same flour split into smaller loaves and kneaded much longer makes better bread. So instead of growing models forever, the field started training smaller models on much more data. And then they noticed something else: once you bake the bread, you serve it to customers for years. A smaller loaf is cheaper to slice every single time someone orders one. So even though it costs more flour-time to knead the smaller loaf, you save money every day forever after. That is why a 2026 8-billion-parameter model trained on 15 trillion tokens beats a 2020 175-billion-parameter model trained on 300 billion tokens.

Key concepts

Transformer architecture

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

6 min: Chinchilla's compute-optimal ratio of 20 tokens-per-param + how it differed from Kaplan + GPT-3 was under-trained + the methodological fix + architectural shape barely changed + post-2022 the field walked PAST optimum + inference cost dominates training cost + Llama-1 to Llama-3 progression + where Chinchilla still applies.

Side-by-side

Model	Year	Params	Tokens	Tokens/Param	Regime
GPT-3	2020	175B	300B	~1.7	Pre-Chinchilla, severely under-trained
Chinchilla	2022	70B	1.4T	20	Compute-optimal
Llama-1 7B	2023	7B	1T	~143	First step past Chinchilla
Llama-2 7B	2023	7B	2T	~285	Walking past Chinchilla
Llama-3 8B	2024	8B	15T	~1875	Far past Chinchilla, inference-optimized
Llama 3.1 405B	2024	405B	15.6T	~39	Past Chinchilla for the frontier dense model

Real-world examples

Real products, models, and research that use this idea.

Hoffmann et al. (Chinchilla, 2022): 70B model on 1.4T tokens beat 175B GPT-3 and 280B Gopher at the same training compute.
Llama-1 7B (Feb 2023): 1T tokens, ratio ~143, deliberately above Chinchilla; demonstrated that small models trained heavily can compete with much larger Chinchilla-optimal models.
Llama-3 8B (Apr 2024): 15T tokens, ratio ~1875, ninety times Chinchilla-optimal; current best small-model benchmark on most tasks.
Llama 3.1 405B (Jul 2024): 15.6T tokens, ratio ~39, still well past Chinchilla, used as the largest dense Llama model.
DeepSeek V3 (Dec 2024): 671B total / 37B active MoE, trained on ~14.8T tokens; uses MoE specifically to keep active parameters small for serving while keeping total parameters large for quality.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy do MoE models change the Chinchilla calculus?

QIf you only have 1e22 training FLOPs, what model size and token count should you pick?

QHow does data quality interact with the Chinchilla scaling law?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Reading Chinchilla as a recipe to FOLLOW. The field followed it for about a year and then deliberately walked past it; modern Llama 3.1 8B trains at ~1875 tokens per param, almost 100x the Chinchilla ratio.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.