Why ramp the learning rate during warmup instead of starting at the peak value?

**Warmup** is a short period at the start of training where the learning rate climbs from near zero up to its peak value. The ramp is usually linear over a few hundred steps. After warmup the schedule typically decays the rate down again, often as a cosine. The reason to ramp instead of starting at the peak is the optimizer itself. Adam, and most of its variants used in fine-tuning, scales each parameter's step by an estimate of how noisy that parameter's gradient has been. The estimate is a running average of squared gradients, called the second moment. At the first step, the optimizer has seen no gradients yet. The second moment is initialised to zero. The bias correction in Adam helps, but the early estimates are still based on very few samples and can be unreliable. The result is that the effective step size in the first few updates can be much larger than intended. If you start at the peak learning rate, those oversized first steps can push the weights far off the pretrained manifold. The loss spikes within a few hundred steps and the run never recovers. Warmup keeps those first steps small while Adam's variance estimate catches up.

**Warmup** exists because Adam needs a few hundred steps to build a reliable variance estimate before taking full-sized updates. The Adam update for parameter theta at step t is: $$\theta_t \leftarrow \theta_{t-1} - \eta_t \cdot \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$$ The denominator is built from `v_t`, the exponential moving average of squared gradients. At step 0 that average is zero. Adam's bias correction multiplies by `1 / (1 - beta_2^t)` to compensate, but with a single sample the correction cannot conjure information that is not there. The effective step in the first few updates is dominated by tiny `v_t` values, so the ratio `m_t / sqrt(v_t)` is unstable and tends to be too large for the actual gradient noise. Multiplying that unstable ratio by a peak learning rate around 1e-5 to 5e-5 still produces an outsized step against pretrained weights that were sitting in a narrow basin. The model jumps out, the loss spikes, and the run is lost. A linear ramp from zero to peak over 3 to 10 percent of total steps (or 100 to 500 fixed steps for short SFT runs) keeps the early multiplier small while `v_t` accumulates enough samples to stabilise. After warmup, the optimizer can apply peak sized steps without the early step instability.