A team is planning a 7B parameter dense model. The Chinchilla heuristic of about 20 tokens per parameter gives the compute optimal token count, but because serving costs dominate their economics, they decide to train on 1.4T tokens instead. Using the approximation C = 6ND, compute: (a) the compute optimal token count under the heuristic, (b) the total training FLOPs for the actual 1.4T token run, and (c) the overtraining multiple relative to optimal.
Chinchilla optimal for 7B is about 140B tokens. Training on 1.4T tokens instead costs C = 6ND, about 5.9e22 FLOPs, a 10x overtrain that pays off when serving cost dominates the model's lifetime economics.
Think of building a delivery van. You could spend a normal amount at the factory and get a van with average fuel economy, or spend ten times longer engineering a smaller, hyper-efficient engine. If the van only drives once, the extra factory time is wasted. But if it will drive a million miles, every mile is cheaper with the efficient engine, and the factory bill becomes a rounding error. Training tokens are the factory time, and serving the model to users is the million miles. Teams overtrain small models far past the textbook optimum because they pay the training bill once but pay the serving bill on every single user request, forever.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This question looks like arithmetic, and the arithmetic is genuinely the point: planning meetings for pretraining runs are full of exactly these back of envelope numbers, and an engineer who fumbles exponents in that room loses the room. But the second half of the question is judgment: the team is choosing to spend ten times the textbook-optimal compute, and you need to explain why that is sane.
We will build both halves: first the two formulas and where they come from, then the calculation done slowly enough to be error-proof, then the economics of overtraining that turned the entire industry away from Chinchilla-optimal training after 2023.
Where C = 6ND comes from
Start with one token flowing through one dense layer. For every weight in that layer, the forward pass performs one multiply and one add, about 2 FLOPs per parameter per token. The backward pass has to compute gradients with respect to both the activations and the weights, which costs roughly two forward passes, another 4 FLOPs per parameter per token. Add them up:
where N is parameter count and D is the number of training tokens. The formula deliberately ignores attention score computation, normalization, activation functions, and optimizer bookkeeping. For typical context lengths those are small corrections; the matmuls against the weight matrices dominate. That is also the formula's known blind spot: attention FLOPs scale with sequence length squared per token pair, so a run at 128K context pays a real surcharge that 6ND never sees.
The companion heuristic comes from the Chinchilla experiments, which fit the loss surface L(N, D) and asked: for a fixed compute budget C, what allocation of N and D minimizes loss? The fitted answer was that parameters and tokens should scale together, landing near D = 20N for the data mixtures tested. A 70B model wants about 1.4T tokens; a 7B model wants about 140B. Both numbers are heuristics over one lab's data distribution, but they have proven durable as planning priors across the industry.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3 8B trained on about 15T tokens, near 1900 tokens per parameter and roughly 90x the Chinchilla heuristic, because Meta optimizes for serving cost across its product fleet.
- Qwen 2.5 trained its dense models on 18T tokens, the same overtraining bet applied to small models that must serve cheaply at scale.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you choose the overtraining multiple rather than defaulting to 10x?
Add expected lifetime inference FLOPs to the objective: minimize training plus serving cost subject to a quality floor, using projected query volume and tokens per request.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Sliding an exponent in C = 6ND, or treating 20 tokens per parameter as a law of nature rather than a heuristic that optimizes training compute only and ignores inference cost entirely.
60 second bullets to scan on the way to the call.
What does each symbol in C = 6ND stand for, and what does the formula deliberately ignore?
How many tokens does the 20 tokens per parameter heuristic assign to a 7B model?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.