Why ramp the learning rate during warmup instead of starting at the peak value?

Attempt it

Adam-family optimizers track a per-parameter variance estimate v_t (the second moment of the gradient). At step 0 that estimate is initialized to zero and has not seen enough samples to be meaningful, so the effective step size, which divides by sqrt(v_t), is unstable and tends to be too large for the actual gradient noise.

A warmup that ramps the LR from near 0 up to peak over a few hundred steps (typically 3 to 10 percent of total steps, or a fixed 100 to 500 steps for short SFT runs) lets v_t collect enough gradient samples to stabilize before the optimizer is allowed to take full-sized steps. Without warmup, the first few updates often push weights far enough off the pretrained manifold that the loss spikes early and the run never recovers.

Your answer

Quick reference

TL;DR

Warmup ramps the learning rate from near zero so Adam's variance estimate can stabilize before peak-sized steps are taken, preventing early loss spikes that the run never recovers from.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture pushing a heavy swing in a garden. If you shove it full force on the first swing while it is still, the chain jerks and the swing flies sideways instead of arcing forward. If you start with a few gentle pushes, the swing finds its rhythm, then your full push lands in time with the motion and the swing carries cleanly. Warmup is the few gentle pushes for a training run. Adam, the trainer most fine-tuning uses, needs a handful of small steps to feel out the slope of the loss before it can take a full-sized one safely. Without those small steps, the first big push lands at the wrong moment and the model goes sideways. The run never finds its rhythm again.

Key concepts

Learning rate warmup Adam optimizer Learning rate schedule Training stability

Detailed answer & concept explanation

~10 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: state the Adam variance estimate behavior at step 0, walk the bias correction limit, link to fine-tuning's narrow basin, set typical warmup ratios, and contrast with optimizers that handle this internally.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face Transformers Trainer exposes warmup_ratio and warmup_steps, with most Llama 4 and Mistral SFT recipes using a 3 to 10 percent warmup ratio.
Axolotl configs default to a linear warmup followed by cosine decay for QLoRA runs on Llama 4 and Gemma 4.
DeepSeek V4 distillation recipes use a 5 percent linear warmup ratio on top of an Adam optimizer with beta_2 set to 0.999.
Unsloth tutorials for fine-tuning small open weight models emphasise warmup steps as a fix for training that loss-spikes in the first few hundred steps.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the value of beta_2 in Adam interact with the required warmup length?

Sign in or go Pro to see the approach for this follow-up.

QWhy does fine-tuning often need warmup even though the pretrained model is already in a good basin?

Sign in or go Pro to see the approach for this follow-up.

QCan you achieve the effect of warmup by simply using a lower peak learning rate?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Skipping warmup or setting it to zero steps because the dataset is small. Adam's variance estimate still needs time to stabilize regardless of dataset size, and skipping warmup is a common cause of early loss spikes.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.