The critical batch size is small early in training and grows as loss falls, so a batch ramp buys more optimizer steps per token early and stops wasting data parallelism later.
Imagine a chef tasting a brand new soup. At the start the soup is far off: one small spoonful is enough to tell it needs salt, so tasting tiny amounts and adjusting often is the fastest way to improve it. Later, when the soup is nearly right, one spoonful might mislead. Now the chef stirs well and tastes several spoonfuls before each careful tweak. Training a model follows the same logic. Early on, the model is so wrong that even a small sample of examples points clearly at what to fix, so it pays to make many quick corrections. As the model improves, each correction becomes subtler, and you need to average over many more examples before you can trust that a fix is real. Starting small and growing the sample keeps every example doing useful work.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Frontier training configs contain a detail that puzzles newcomers: the global batch size is not constant. Llama 3 stepped from 4M to 16M tokens; DeepSeek-V3 ramped from 3072 to 15360 sequences across its first 469B tokens. If big batches are good, why not start big? If small batches are good, why not stay small?
The answer is that neither is good in itself. A fixed token budget buys a product of two factors, number of optimizer steps times tokens per step, and the optimal split between them moves over the course of training. Understanding why it moves, and how labs track it, is the core of this question.
Steps versus width: what a token budget actually buys
Fix a budget of T tokens. Train at batch size B and you get T / B optimizer steps. Double B and you halve the step count. The question is when a wider, less noisy gradient is worth sacrificing half your updates.
Think of each batch gradient as the true gradient plus zero mean noise whose variance shrinks like 1 / B. When the noise term dominates the true gradient, averaging over more samples directly improves the direction you step in, and width is worth buying. When the true gradient already dominates, extra width polishes an estimate that was good enough, while costing steps that would each have made real progress.
This is why the question is about an exchange rate rather than a fixed best value. Early in training, loss gradients are enormous: the model assigns near uniform probability over the vocabulary and almost any direction helps. Late in training, the true gradient is tiny while per sample variance remains substantial. The same 4M token batch that was wastefully wide at step zero becomes noisily narrow a trillion tokens later.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3 stepped its global batch upward in stages during the 405B run, from 4M toward 16M tokens, the exact pattern this question describes.
- DeepSeek-V3 ramped batch size from 3072 to 15360 sequences over the first 469B tokens of its run, documented in the technical report.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you measure the critical batch size during a live run?
Compare gradient norms computed at two different accumulation settings; the gap estimates the noise to signal ratio that defines the noise scale.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Explaining the ramp with hardware stories, like memory pressure or network warmup, when the real driver is optimization: gradient noise relative to signal changes as the loss falls.
60 second bullets to scan on the way to the call.
What does the critical batch size separate, and which side wastes compute?
Why is the gradient estimate from a small batch sufficient early in training?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.