Which hyperparameter is most likely when loss spikes the moment warmup ends?
A LoRA fine-tune trains cleanly through 200 warmup steps, then the moment warmup finishes and the schedule hits its plateau, the loss jumps sharply and never recovers. Name the most likely knob, explain in one or two sentences why this exact failure mode points there, and describe the first remediation step.
Peak learning rate set too high for the regime. Warmup masks it; the moment the schedule reaches plateau, updates push the parameters out of their basin.
Imagine pushing a kid on a swing. You start with tiny taps so they get used to the rhythm, then settle into a steady push. If your steady push is way too hard, the swing flies up the chains and the kid bails out. The tiny early taps felt fine, but the steady push was always going to be too strong. You just could not see it during the gentle start. Training a model with warmup works the same way. The early steps are deliberately weak so the model can settle in. When the schedule hands over to the full peak rate, that rate is what your run experiences. If somebody copied a number from a different recipe, the gentle warmup gave no warning, and the plateau is where the wheels come off.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A loss curve that descends smoothly through warmup and then spikes the instant the schedule reaches its plateau is one of the most diagnostically clear failure modes in fine-tuning. The geometry of the warmup to peak handoff tells you almost everything you need to know about which variable is responsible. The trick is to read it correctly and resist the urge to chase the more glamorous suspects.
Warmup is a deliberate suppression mechanism. The optimizer's nominal peak learning rate is the value you set in the config. During the warmup phase, the effective rate is a fraction of that value. It ramps linearly from near zero up to the peak over a fixed number of steps. The point of warmup is to let the optimizer's adaptive estimates settle. It also lets layer norm and embedding statistics catch up to the new task before any sizable parameter changes happen. While that ramp is in progress, the run trains below the configured peak.
This explains why a peak just above the stable limit hides itself during warmup. The updates stay below that limit until the ramp nears its top. The top of the ramp is the first time the optimizer sees the full peak. If the peak is wrong for the regime, the full-strength steps there are where the run goes off the rails. The signature: clean descent through warmup, then a sharp jump around step N, where N is the warmup length. The loss never returns to anything like its pre-spike level.
What warmup does to the effective LR
A standard linear warmup schedule multiplies the configured peak learning rate by a factor that grows from zero up to one over the warmup window. At step t inside a warmup of length W, the effective rate is:
For a peak of 5e-4 and a warmup of 200 steps, the effective rate at step 1 is 2.5e-6 and at step 100 is 2.5e-4. The full 5e-4 only engages at step 200 and stays there for the plateau portion of the schedule. At step 199 the rate is already 4.975e-4, 99.5 percent of peak, so the LR does not jump at the boundary. The spike appears where the rate first crosses the stable limit. For a peak just above that limit, that point is the top of the ramp.
The consequence is that during warmup the parameters move slowly and the loss curve looks reassuring. Early in warmup, a peak slightly above the stable limit looks the same as a correctly tuned peak, because both run at small effective rates. A peak far above the limit crosses it earlier, partway through the ramp. The only thing that changes at the end of warmup is which configured peak the schedule transitions to. That value controls everything afterward.
This is why the diagnostic geometry is so clean. A spike that lines up with the end of warmup cannot easily be blamed on data quality, on precision artifacts, or on a numerical bug. None of those would line up with the schedule. They tend to produce spikes at random steps or to show up as gradual instability. A spike timed to the top of the ramp points at the variable the ramp controls, which is the effective learning rate.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Unsloth's LoRA notebooks use a peak learning rate of 2e-4; the same number in a full fine-tune of an 8B model is about ten times the usual full fine-tuning range.
- TRL's DPOConfig defaults to a learning rate of 1e-6, so an SFT-scale 1e-5 left in place for a DPO run is ten times the default.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does Adam's optimizer state make a too-high peak worse?
Warmup makes the learning rate small, not the gradients. Adam's normalized step is about one learning rate per coordinate, so the peak sets the step size directly. Once a full-strength step lands in a high-loss region, gradients grow. The second-moment average (beta2 = 0.999) lags behind while the first moment follows quickly, so the next steps are larger still.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Blaming the scheduler or the dataset. The scheduler is doing exactly what it was told. The peak value it was told to ramp to is the actual problem.
60 second bullets to scan on the way to the call.
Effective LR during warmup versus at peak
Why divergence locked to the plateau handoff points at the peak value
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.