Why ramp the learning rate during warmup instead of starting at the peak value?
Warmup ramps the learning rate from near zero so Adam's variance estimate can stabilize before peak-sized steps are taken, preventing early loss spikes that the run never recovers from.
Picture pushing a heavy swing in a garden. If you shove it full force on the first swing while it is still, the chain jerks and the swing flies sideways instead of arcing forward. If you start with a few gentle pushes, the swing finds its rhythm, then your full push lands in time with the motion and the swing carries cleanly. Warmup is the few gentle pushes for a training run. Adam, the trainer most fine-tuning uses, needs a handful of small steps to feel out the slope of the loss before it can take a full-sized one safely. Without those small steps, the first big push lands at the wrong moment and the model goes sideways. The run never finds its rhythm again.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Learning rate warmup is one of those settings every fine-tuning recipe carries forward but few practitioners can defend mechanically. The default is somewhere between three and ten percent of total steps for a linear ramp, the warmup ratio is buried in the config, and most people never touch it again. That works because the default is well chosen, but it leaves the question of why warmup exists at all unanswered.
The short answer is that Adam, the optimizer almost every fine-tuning run uses, has a known instability in its first few steps. The longer answer involves the second-moment buffer, how the bias correction fails to fully fix the early estimate, and why fine-tuning is especially fragile to oversized early updates compared to pretraining.
This deep dive walks through Adam's update rule, examines what happens to the variance estimate at step zero, links that behavior to the narrow loss basin a pretrained model sits in, and explains why the standard linear ramp solves the problem cleanly. By the end the warmup ratio should look like a consequence of optimizer mechanics rather than folklore.
Adam's update rule and the second-moment buffer
Adam tracks two running averages per parameter. The first moment m_t is an EMA of past gradients, smoothed with beta_1 (default 0.9). The second moment v_t is an EMA of past squared gradients, smoothed with beta_2 (default 0.999). The update applied at step t is:
where the hats denote bias-corrected versions of m_t and v_t. The bias correction divides by 1 - beta^t to account for the EMAs being initialised at zero and therefore biased low at small t.
What v_t represents
The second moment estimates the magnitude of recent gradients. Dividing by its square root normalises the update so that noisy parameters take smaller steps and stable parameters take larger ones. This is the core idea of adaptive optimizers: scale step size per parameter based on observed gradient noise.
What happens at step zero
At initialisation, both m_0 and v_0 are zero. At step 1, the update is (1 - beta_1) * g_1 for the first moment and (1 - beta_2) * g_1^2 for the second moment. With beta_2 = 0.999, the second moment after one step is 0.001 * g_1^2, a thousand times smaller than the gradient magnitude. The bias correction multiplies this by 1 / (1 - beta_2^1) = 1000, bringing it back to g_1^2.
Mathematically the correction restores the magnitude, but it cannot restore information. The estimate v_t after one or two steps is based on a single squared gradient, which has very high variance as an estimator of the true second moment. The effective step direction can be off, and the per-coordinate scaling can over- or under-shoot dramatically.
This early-step instability is the core problem warmup addresses.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Hugging Face Transformers Trainer exposes warmup_ratio and warmup_steps, with most Llama 4 and Mistral SFT recipes using a 3 to 10 percent warmup ratio.
- Axolotl configs default to a linear warmup followed by cosine decay for QLoRA runs on Llama 4 and Gemma 4.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the value of beta_2 in Adam interact with the required warmup length?
Higher beta_2 means a longer effective averaging window, so the second-moment buffer takes more steps to populate; the warmup length should grow accordingly. Quantify with the half-life formula for an EMA.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Skipping warmup or setting it to zero steps because the dataset is small. Adam's variance estimate still needs time to stabilize regardless of dataset size, and skipping warmup is a common cause of early loss spikes.
60 second bullets to scan on the way to the call.
Why Adam needs warmup at the start of training
How the second-moment buffer behaves at step zero
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.