A pretraining run crashes at step 120000. The resume restores model weights and full optimizer state from the checkpoint, but the data loader restarts with a fresh shuffle seed and no record of its previous position. Predict (a) what happens to the loss curve immediately at resume, and (b) how the resumed run compares to a hypothetical uninterrupted run over the next 50B tokens.
No visible spike because optimizer state survived, but the data stream silently diverges: some shards repeat, others are skipped, and the run is no longer the run you designed.
Imagine you are reading a giant series of books for school, taking notes in a notebook. You drop everything on day 60 and pick up later. If you saved your notebook, you keep your understanding. But if you forgot which book you were on and grabbed a random one from the shelf, you might re-read books you already finished and skip books you have not seen. Your overall reading still feels like learning, your notes still grow, and you still pass the final exam. But your reading log is wrong: you read some books twice and never opened others. If the curriculum carefully balanced subjects, that mistake quietly shifts what you actually learned, even though no single day felt off.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This scenario is one of the cleanest demonstrations that a clean-looking loss curve does not mean a clean run. The behaviour after resume is statistically silent on the dashboard while being structurally wrong in the underlying data contract. Senior pretraining engineers learn this once, usually the hard way, and then treat data-loader state as a non-optional checkpoint artifact thereafter.
This deep dive separates the visible and silent failure modes of resume, explains why this scenario falls into the silent bucket, traces the statistical drift to its consequences for unique-token counts and mixture accounting, and closes with the complete checkpoint contract that prevents the bug.
Why optimizer state controls the spike
Adam carries two moment estimates per parameter: a first moment that smooths the gradient direction and a second moment that scales it by recent magnitude. After many steps these moments have stabilized to encode the optimization trajectory. They are what makes Adam different from SGD, and they are what causes resume spikes when they are missing.
If weights are restored but optimizer state is missing, the first few steps effectively run with reinitialized moments. The first-moment buffer takes time to refill; the second-moment buffer divides every update by very small numbers initially. Step sizes go wrong in non-obvious directions, gradient clipping fires, and the loss spikes or drifts upward for thousands of steps.
In this scenario, optimizer state was preserved. The moments are intact, the learning rate schedule is intact, and the model picks up where it left off. The curve looks fine because the optimization machinery is fine. The bug lives somewhere else.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3 and OLMo 2 training reports both emphasize data-loader state in their checkpoint format and resume procedures.
- Megatron-LM and NeMo frameworks expose data-loader state checkpointing as a first-class feature for exactly this reason.
What an interviewer would ask next. Try answering before peeking at the approach.
QYour loss curve at resume actually shows a small spike instead of a clean continuation. What is your first hypothesis?
Optimizer state likely partially restored or scaler state for mixed precision is missing. Less likely is data drift; data-shape changes usually cause local blips, not spikes. Verify checkpoint format completeness.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming a clean-looking loss curve after resume means the resume was correct. Loss is insensitive to the data drift this bug causes.
60 second bullets to scan on the way to the call.
What causes a loss spike at resume?
Why does this scenario not produce a spike?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.