A LoRA fine-tune trains cleanly through 200 warmup steps, then the moment warmup finishes and the schedule hits its plateau, the loss jumps sharply and never recovers. Name the most likely knob, explain in one or two sentences why this exact failure mode points there, and describe the first remediation step.
Free · 2 AI evals / day
**Peak learning rate** is the prime suspect. Warmup ramps the effective LR from zero up to a peak over the first few hundred steps. While that ramp is in progress, the effective rate is small, so even a wildly oversized peak feels safe.
The moment warmup finishes, the optimizer hands the full peak rate to the parameter updates. If that peak is too large for the regime you are in, the updates take steps too big for the local loss surface. The parameters jump out of the basin they were sitting in, the optimizer momentum points in a stale direction, and the loss spikes and never recovers.
The usual root cause is copying a learning rate from a different recipe. Full fine-tuning, LoRA, and DPO all sit at different safe ranges, and a default that works in one breaks the others.
The first fix is to halve the peak learning rate and rerun. If the spike disappears, sweep down from there to find the largest stable value. Lengthening warmup may delay the spike, but it does not fix the underlying mismatch.
**Peak learning rate is too high** for the regime. The signature is exactly what was described: clean training through warmup, then a sharp loss spike the instant the schedule reaches its plateau.
The mechanism is straightforward. During warmup the effective LR is a small fraction of the configured peak, so updates are gentle and the loss declines smoothly. The schedule hands control to the peak rate at the end of warmup, and that peak is what the run actually trains under from then on. If the peak is calibrated for a different regime, the first full-strength step takes parameters out of their current basin.
The usual cause is a mis-anchored default. Full fine-tune LRs of 1e-5 to 5e-5 are far too small for LoRA, which often wants 1e-4 to 5e-4. Conversely, a LoRA-scale 5e-4 dropped into a full fine-tune blows up. DPO and other preference-tuning losses are even more sensitive, with safe peaks closer to 5e-6.
First remediation: halve the peak learning rate and rerun for a few hundred steps past the previous spike point. If the spike is gone, sweep downward to find the largest stable value. Lengthening warmup is a useful secondary mitigation but does not fix a mis-scaled peak.
Quick reference
TL;DR
Peak learning rate set too high for the regime. Warmup masks it; the moment the schedule reaches plateau, updates blow the parameters off the loss surface.
Imagine pushing a kid on a swing. You start with tiny taps so they get used to the rhythm, then settle into a steady push. If your steady push is way too hard, the swing flies up the chains and the kid bails out. The tiny early taps felt fine, but the steady push was always going to be too strong, you just could not see it during the gentle start. Training a model with warmup works the same way. The early steps are deliberately weak so the model can settle in. When the schedule hands over to the full peak rate, that rate is what your run actually experiences. If somebody copied a number from a different recipe, the gentle warmup gave no warning, and the plateau is where the wheels come off.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
The spike does not coincide with the end of warmup but appears randomly mid-run Random mid-run spikes point at bad data, precision overflow, or an outlier batch, not at the peak LR magnitude.
Halving the peak LR twice still produces the same loss explosion At that point the peak is no longer the load-bearing variable; look at dataset corruption, label leakage, or precision issues.
Loss climbs slowly over many steps after warmup rather than jumping in one step A slow drift suggests overfitting or wrong regularisation; the peak-LR hypothesis predicts a sharp, locked-in jump.
Quick scan · 2–4 min
Everything important, quickly.
5 min: warmup masks the peak + plateau handoff is the actual rate + regime-specific safe ranges + Adam state staleness + halve and rerun as first fix + longer warmup as secondary mitigation.
Real-world examples
Real products, models, and research that use this idea.
1Llama 4 Maverick LoRA recipes in TRL and Unsloth default to peak LRs around 1e-4 to 2e-4; the same number plugged into a full fine-tune of the base reliably reproduces a post-warmup spike.
2DPO recipes on Claude Opus 4.7-style preference data routinely sit at 5e-7 to 5e-6 peak; an SFT-tuned 1e-5 dropped in causes a textbook divergence one step after warmup ends.
3Mistral and Qwen 3.5 QLoRA cookbooks set peak LR around 2e-4 with 3 to 10 percent warmup, precisely because higher peaks tip over at the plateau.
4Hugging Face TRL release notes call out the regime mismatch in the SFTTrainer to DPOTrainer migration, where users see the spike when they leave the SFT LR in place.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does Adam's optimizer state become stale at the moment the peak engages?
Sign in or go Pro to see the approach for this follow-up.
QIf halving the peak LR helps but still leaves instability, what is your next move?
Sign in or go Pro to see the approach for this follow-up.
QWhy does DPO sit two orders of magnitude below SFT on safe peak LR?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Blaming the scheduler or the dataset. The scheduler is doing exactly what it was told. The peak value it was told to ramp to is the actual problem.