A 13B model starts pretraining with global batch 4M tokens and peak LR 3e-4 on a cosine schedule. The config intended warmup_steps=2000, but a units bug set warmup_steps=20, so the LR hits its full peak almost immediately after initialization. Predict what the loss curve and the gradient norm chart look like over the first few hundred steps.
Peak LR applied to a near-random initialization produces exploding gradient norms and a spiking or diverging loss within the first few hundred steps.
Imagine you are learning to ride a bike. A good teacher starts you on a gentle slope at walking pace, then gradually lets you go faster as you find your balance. That gradual speedup is warmup. Now imagine on day one the teacher straps you to a motorcycle at full throttle. You have no skill yet, the throttle is wide open, and the most likely outcome is a crash within seconds. That is what hitting peak learning rate in 20 steps does to a freshly initialized 13B model. The weights are essentially random, the loss surface is chaotic at that point, and full-size optimizer steps fling the weights into regions the model cannot recover from. Gradient norms blow up, loss spikes, and divergence is likely.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Warmup is one of those training-recipe choices that looks like a minor hyperparameter on paper and turns out to be load-bearing in practice. The question describes a small but realistic configuration bug: warmup_steps set to 20 instead of 2000, a hundredfold compression of the ramp. The intuition test is whether you can predict the resulting behaviour without running it.
The answer is unambiguous at this scale. A 13B model at a 4M-token batch with 3e-4 peak LR and a 20-step warmup will go unstable in the first few hundred steps. The reasons are not folklore; they are mechanical consequences of how loss landscapes look at initialization and how Adam moment estimates behave when they have not had time to converge.
This deep dive walks through the three coupled mechanisms, the resulting dashboard signatures, and why modern frontier recipes converged on multi thousand step warmups at this scale.
Why warmup matters at initialization
Right after initialization, model weights are drawn from a calibrated random distribution (typically a scaled-variance Gaussian per the Xavier or He scheme, or a layer-norm-aware variant). The forward pass produces well-scaled activations on the first batch, but the gradients from a single backward pass are noisy and reflect the structure of one mini-batch on a model that knows nothing.
The loss surface in that region is high-curvature: many directions have large second derivatives, and the optimizer's linear approximation breaks down for any update larger than a small step. A second-order view says the effective trust region is small, and warmup is the mechanism that keeps step magnitudes inside it. As training progresses and weights move into a better-conditioned basin, the trust region widens and full-size steps become safe.
Applying peak LR at step 1 violates the trust region by orders of magnitude. The optimizer's local model of the loss surface is wrong, the resulting updates push weights into regions where the loss is higher than where they came from, and the next batch's gradients are even less reliable. The process compounds.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3 uses 8000 steps of warmup for its 405B model at this approximate batch size and peak LR
- DeepSeek-V3 documents a 2000-step warmup phase as a default for its 671B MoE pretraining recipe
What an interviewer would ask next. Try answering before peeking at the approach.
QIf you only noticed the bug after the run was already 500 steps in, what would you do?
Discuss whether to kill versus try to recover. Most likely outcome is the loss has already spiked and the weights are far enough off the stable manifold that a full restart from initialization is cheaper than trying to anneal back.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Predicting a smooth loss curve because the LR schedule looks reasonable on paper, missing that the first few thousand steps are exactly where warmup matters most.
60 second bullets to scan on the way to the call.
Why warmup matters at initialization specifically
How Adam moment estimates interact with early training stability
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.