What does a 20 step warmup do to a 13B run?

Predict output·Medium·4.0 · 0·~2 min·Asked atAlibaba Doordash Uniphore

Attempt it

A 13B model starts pretraining with global batch 4M tokens and peak LR 3e-4 on a cosine schedule. The config intended warmup_steps=2000, but a units bug set warmup_steps=20, so the LR hits its full peak almost immediately after initialization. Predict what the loss curve and the gradient norm chart look like over the first few hundred steps.

The run will be unstable almost immediately. **Warmup** ramps the learning rate up from near zero over the first several thousand steps. It exists because a freshly initialized model has weights drawn from a random distribution, and the loss surface near that point is chaotic. Also, the moving averages that the Adam optimizer keeps (its moment estimates) have not had time to calibrate. Full-size optimizer steps from a random init produce huge, badly aimed updates. With a 20-step warmup, the learning rate hits its full 3e-4 peak before the model has any chance to stabilize. Expect gradient norms to climb sharply within the first tens of steps, often into the hundreds or thousands. Expect the loss to spike up after a brief initial drop, or to outright diverge to NaN (not a number). Gradient clipping will fire on most steps, which is a secondary symptom rather than a fix. The correct prediction is: exploding gradient norms early, a spiking or diverging loss curve, and a run that needs to be killed and restarted with the intended 2000-step warmup.

The run will go unstable inside the first few hundred steps. Two coupled things are happening. First, the model weights start near random initialization. The loss surface in that region is chaotic, with steep curvature and ill-conditioned Hessian structure. Updates of size proportional to peak LR push the weights through the curvature faster than they can adapt. This produces huge increases in activation magnitude and gradient norm. Second, Adam's first and second moment estimates `m_t` and `v_t` start at zero and take many steps to converge to their true running averages. Until they do, the effective step size per coordinate is poorly calibrated. Warmup partly compensates for this. It keeps the LR small while the moments warm up, so the update magnitudes stay sane. Compressing warmup from 2000 steps to 20 strips both protections. Expect: - **Gradient norms** climbing into the hundreds or thousands within the first few dozen steps, with the clip rate pinned near 100 percent. - **Loss curve** dropping briefly as the model absorbs trivial token statistics, then spiking upward, then either oscillating wildly or going to NaN. - **Activation norms** growing layer by layer, with attention logits saturating into one-hot softmax outputs. Clipping at 1.0 might delay the divergence by a few hundred steps, but it does not save the run. The trajectory is already off the manifold of stable solutions. Kill, fix the config, restart from initialization.

TL;DR

Peak LR applied to a near-random initialization produces exploding gradient norms and a spiking or diverging loss within the first few hundred steps.

Memory aid

Easy to grasp

Imagine you are learning to ride a bike. A good teacher starts you on a gentle slope at walking pace, then gradually lets you go faster as you find your balance. That gradual speedup is warmup. Now imagine on day one the teacher straps you to a motorcycle at full throttle. You have no skill yet, the throttle is wide open, and the most likely outcome is a crash within seconds. That is what hitting peak learning rate in 20 steps does to a freshly initialized 13B model. The weights are close to random, and the loss surface is chaotic at that point. Full-size optimizer steps fling the weights into regions the model cannot recover from. Gradient norms blow up, loss spikes, and divergence is likely.

Key concepts

Pretraining

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Warmup looks like a minor hyperparameter on paper and turns out to be a key part of the training recipe in practice. The configuration bug is small but realistic: warmup_steps set to 20 instead of 2000, a hundredfold compression of the ramp.

A 13B model at a 4M-token batch with 3e-4 peak LR and a 20-step warmup will go unstable in the first few hundred steps. The reasons are not folklore. They are mechanical consequences of how loss landscapes look at initialization and how Adam moment estimates behave when they have not had time to converge.

Why warmup matters at initialization

Right after initialization, model weights are drawn from a calibrated random distribution (typically a scaled-variance Gaussian per the Xavier or He scheme, or a layer-norm-aware variant). The forward pass produces well-scaled activations on the first batch. But the gradients from a single backward pass are noisy. They reflect the structure of one mini-batch on a model that knows nothing.

The loss surface in that region is high-curvature: many directions have large second derivatives. The optimizer's linear approximation breaks down for any update larger than a small step. A second-order view says the effective trust region is small, and warmup is the mechanism that keeps step magnitudes inside it. As training progresses and weights move into a better-conditioned basin, the trust region widens and full-size steps become safe.

Applying peak LR at step 1 violates the trust region by orders of magnitude. The optimizer's local model of the loss surface is wrong. The resulting updates push weights into regions where the loss is higher than where they came from. The next batch's gradients are then even less reliable. The process compounds.

Adam moment bias is the second compounding factor

What the dashboard will show

Why published recipes use warmups of thousands of steps

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Llama 3 405B used a linear warmup of 8,000 steps to a peak LR of 8e-5, with a batch that started at 4M tokens and later grew to 16M
DeepSeek-V3 (671B MoE) raised its LR linearly from 0 to 2.2e-4 over the first 2,000 steps

What an interviewer would ask next. Try answering before peeking at the approach.

QIf you only noticed the bug after the run was already 500 steps in, what would you do?

Discuss whether to kill versus try to recover. The most likely outcome is that the loss has already spiked. The weights are then far enough off the stable manifold that a full restart from initialization is cheaper than trying to anneal back.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Predicting a smooth loss curve because the LR schedule looks reasonable on paper. The first few thousand steps are where warmup matters most.

60 second bullets to scan on the way to the call.

Why warmup matters at initialization specifically
How Adam moment estimates interact with early training stability

Primary sources. Browse if you want the original framing.