Loss spikes the moment warmup ends: which hyperparameter is the prime suspect?
A LoRA fine-tune trains cleanly through 200 warmup steps, then the moment warmup finishes and the schedule hits its plateau, the loss jumps sharply and never recovers. Name the most likely knob, explain in one or two sentences why this exact failure mode points there, and describe the first remediation step.
Peak learning rate set too high for the regime. Warmup masks it; the moment the schedule reaches plateau, updates blow the parameters off the loss surface.
Imagine pushing a kid on a swing. You start with tiny taps so they get used to the rhythm, then settle into a steady push. If your steady push is way too hard, the swing flies up the chains and the kid bails out. The tiny early taps felt fine, but the steady push was always going to be too strong, you just could not see it during the gentle start. Training a model with warmup works the same way. The early steps are deliberately weak so the model can settle in. When the schedule hands over to the full peak rate, that rate is what your run actually experiences. If somebody copied a number from a different recipe, the gentle warmup gave no warning, and the plateau is where the wheels come off.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A loss curve that descends smoothly through warmup and then spikes the instant the schedule reaches its plateau is one of the most diagnostically clear failure modes in fine-tuning. The geometry of the warmup to peak handoff tells you almost everything you need to know about which variable is responsible. The trick is to read it correctly and resist the urge to chase the more glamorous suspects.
Warmup is a deliberate suppression mechanism. The optimizer's nominal peak learning rate is the value you set in the config, but during the warmup phase the effective rate is a small fraction of that value, ramping linearly from near zero up to the peak over a fixed number of steps. The point of warmup is to let the optimizer's adaptive estimates settle and to let layer norm and embedding statistics catch up to the new task before any sizable parameter changes happen. While that ramp is in progress, the run is effectively training at a tiny learning rate, regardless of what the configured peak says.
This explains why a too-high peak hides itself during warmup. The updates are simply too small to do damage, even if the configured peak is grossly mis-scaled. The plateau handoff is the first moment the optimizer sees the full peak. If the peak is wrong for the regime, that single first full-strength step is where the run goes off the rails. The signature is unmistakable: clean descent up to step N, sharp jump at step N+1 where N is the warmup length, and the loss never returns to anything like its pre-spike level. This deep dive walks through why this pattern is so reliable, what the regime-specific safe ranges look like, why Adam's bookkeeping makes recovery effectively impossible, and the order in which you should attack the problem.
What warmup actually does to the effective LR
A standard linear warmup schedule multiplies the configured peak learning rate by a factor that grows from zero up to one over the warmup window. At step t inside a warmup of length W, the effective rate is:
For a peak of 5e-4 and a warmup of 200 steps, the effective rate at step 1 is 2.5e-6 and at step 100 is 2.5e-4. The full 5e-4 only engages at step 200 and stays there for the plateau portion of the schedule.
The consequence is that during warmup the parameters move slowly and the loss curve looks reassuring. A peak that is wildly out of regime is indistinguishable in this phase from a peak that is correctly tuned. Both are running at the same tiny effective rates. The only thing that changes at the end of warmup is which configured peak the schedule transitions to, and that is the value that controls everything afterward.
This is why the diagnostic geometry is so clean. A spike locked to the exact step where warmup ends cannot easily be blamed on data quality, on precision artifacts, or on a numerical bug, because none of those would line up with a configuration boundary. They tend to produce spikes at random steps or to manifest as gradual instability. A boundary-locked spike points at the variable that changes at that boundary, which is the effective learning rate.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 4 Maverick LoRA recipes in TRL and Unsloth default to peak LRs around 1e-4 to 2e-4; the same number plugged into a full fine-tune of the base reliably reproduces a post-warmup spike.
- DPO recipes on Claude Opus 4.7-style preference data routinely sit at 5e-7 to 5e-6 peak; an SFT-tuned 1e-5 dropped in causes a textbook divergence one step after warmup ends.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does Adam's optimizer state become stale at the moment the peak engages?
Adam's second-moment estimate accumulates under the tiny warmup gradients, so the preconditioner reflects that small-step regime. The first full-strength update lands in a region whose curvature was not represented in the running averages, so the adaptive scaling no longer protects against oversized steps.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Blaming the scheduler or the dataset. The scheduler is doing exactly what it was told. The peak value it was told to ramp to is the actual problem.
60 second bullets to scan on the way to the call.
Effective LR during warmup versus at peak
Why divergence locked to the plateau handoff points at the peak value
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.