Order the on call response to a sudden loss spike at 600B tokens
- 1File the incident with the bad shard IDs so the data team can audit them
- 2Rewind: restore model, optimizer, and data loader state from the last healthy checkpoint
- 3Resume training and watch gradient norms closely for a recurrence
- 4Skip the suspect data window so the run does not replay the same batches
- 5Identify the data window the run consumed in the steps just before the spike
- 6Confirm the spike on the dashboard: loss, gradient norm, and per domain eval curves around the offending step
Confirm the spike on telemetry, identify the data window, rewind to a healthy checkpoint, skip the suspect batches, resume with monitoring, then file the incident.
Think of a long road trip with a logbook. The car jolts once. You first check the dashboard to confirm what just happened. Then you note which stretch of road you were on. Next you back up to the last rest stop where the car was fine. You take a detour around the suspect stretch and keep driving, watching the gauges carefully. Once you reach the next safe stop, you write a clear report so the maintenance team can examine the road. Skipping the diagnosis or restart steps would mean you either drive blind or replay the same problem and lose more time. The order is doctor then mechanic then paperwork, in that exact sequence.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A loss spike at 600 billion tokens into a serious pretraining run is one of the higher-stakes pages an on-call engineer can take. The model has consumed millions of dollars of compute. Every wrong response costs more compute. The right response is not improvisation; it is a written playbook executed in order.
The six steps in this question, in this order, are the consensus playbook used by frontier labs. Each step does one specific thing, and reordering them produces predictable failure modes: skipping diagnosis means responding to a logging glitch with a full rewind, skipping identification means resuming into the same bad data, skipping the file step means the next on-call has no record and the data team has no shard IDs to audit.
This deep dive walks through each step, what it produces, and what breaks if it is skipped or reordered.
Diagnose: confirm the spike before anything else
The first step is to confirm the page is real. Read loss together with gradient norm, clip rate, and per-domain eval curves at the offending step and the steps around it. A spike that shows up only on loss but not on any other metric is suspicious; a spike that shows up across loss, gradient norm, and per-domain evals is real.
This step protects the rest of the playbook. A meaningful fraction of paged spikes in a 24x7 training operation are dashboard refresh glitches, evaluation-harness drift, or logging-pipeline issues. Triggering a rewind on a fake spike costs hours of compute and pollutes the run timeline with a spurious incident. Capture screenshots of the relevant panels; they become the artifact you attach to the eventual incident report.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3 pretraining used this exact rewind and skip protocol across the 405B run, documented in the technical report
- OLMo 2 from AI2 publishes a stability log that walks through several spike and skip events using this ordering
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat should the post-resume monitoring plan actually look like?
Specify the time window, the metrics (gradient norm, clip rate, attention logit max, per-domain loss), the alert thresholds, and the rollback criterion if a second spike fires within the window.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Restarting the run before identifying the bad data window, then watching the same spike replay because the loader cursor lands on the same shards.
60 second bullets to scan on the way to the call.
Why dashboard confirmation precedes any rewind decision
What state must be restored on rewind for a deterministic resume
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.