Defend the rewind and skip playbook over a permanent LR cut
Across a 400B token stretch, your run hits three loss spikes. The on call playbook says: rewind to the last good checkpoint, skip the offending data window, resume. A senior engineer argues you should instead permanently lower the peak learning rate. Defend the playbook as the correct first response, and state the evidence that would flip you to the LR cut.
Rewind and skip is cheap and targeted; an LR cut taxes every remaining token. Flip to the cut only when evidence shows spikes are systemic, not incidental.
Imagine a long-distance driver who hits a pothole three times on a 400-mile stretch. The first response is not to drive slower for the rest of the trip; the first response is to back up, go around the pothole, and continue at the speed you planned. Slowing down for the whole rest of the trip costs you on every mile, while the pothole cost you only a short detour. You only switch to slowing down if you start hitting potholes everywhere, not just in three specific spots. That broader pattern means the road itself is bad, not the spots. Then the slower speed earns its cost.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Loss spikes are stressful and tempt a defensive response. The defensive response, almost always, is to cut the learning rate. The defensive response is almost always wrong as the first move, because it pays a real cost on every remaining token to suppress something that is often a local coincidence between a specific data window and the optimizer's current state.
This deep dive defends the rewind and skip playbook as the correct first response, explains the bad batch times state mechanism that justifies it, lays out the evidence that would flip the diagnosis to systemic instability, and walks the systemic mitigations that should be considered before reaching for the LR cut alone.
Cost asymmetry over the remaining budget
The two interventions tax different parts of the run. Rewind and skip pays for the rewound steps, full stop. If the playbook rewinds 500 steps and skips 200 batches, the cost is 700 batches of compute, plus the wall-clock cost of the rewind. The rest of the run continues with the originally planned learning rate schedule, which is the schedule the recipe was designed around.
The permanent LR cut pays a different tax. Every remaining step uses a smaller learning rate than the recipe planned, which means each step makes less progress. Across the next 400B-plus tokens, the integrated cost is a measurably worse final loss. The PaLM report and several subsequent studies document this trade-off explicitly: a 10 to 30 percent LR cut typically costs measurable benchmark performance at the end of the run.
The asymmetry is what justifies the playbook. You pay a bounded cost to test the cheap hypothesis. Only if the cheap hypothesis fails do you upgrade to the expensive intervention.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- PaLM's training report documents rewind and skip as the standard response to transient spikes, with bad batch times state interaction as the proposed mechanism.
- OLMo 2 and Llama 3 reports describe spike handling that prefers targeted skips over LR changes.
What an interviewer would ask next. Try answering before peeking at the approach.
QAfter three rewind and skip cycles, gradient norms are stable between spikes but clip rates are slowly climbing. What is your call?
Climbing clip rates without climbing gradient norms suggests the clip threshold is becoming binding more often, often a sign of slowly worsening logit growth. Lean toward z-loss or qk-layernorm before reaching for an LR cut.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reaching for a permanent LR cut after a single spike, paying the cost across the entire remaining run when a targeted skip would have cleared the problem.
60 second bullets to scan on the way to the call.
Why is rewind and skip cheap relative to an LR cut over a long remaining budget?
What is the bad batch times optimizer state interaction theory of spikes?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.