- 1Training step counter and bookkeeping (epoch index, global step, gradient accumulation micro step counter)
- 2Learning rate scheduler state (so warmup/decay continues from the right step, not step 0)
- 3Data sampler / DataLoader iterator position (so the very next batch is the one the run would have seen)
- 4RNG state across Python, NumPy, PyTorch CPU, and CUDA (so dropout, augmentations, and shuffles are reproducible)
- 5Optimizer state (Adam moments m and v, plus the global step counter)
- 6Model weights (the parameters themselves)
Restore weights, then optimizer state, then LR scheduler, then RNG, then data iterator, then step counters. Each missing layer breaks bit-exact resume in a different way.
Imagine pausing a long board game and coming back tomorrow. The pieces on the board are the model weights. The notes your players wrote about how aggressively each one was playing are the optimizer state. The timer showing how far into the tournament you are is the scheduler. The shuffled deck and the dice memory are the RNG. The exact spot in the playlist of question cards is the data iterator. And the scorecard with round numbers is the bookkeeping. If you reload only the pieces and lose the rest, the game continues but it is not the same game. Bit-exact resume means restoring every single one of those, in the right order, so the next move is identical to what tomorrow would have been.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A training checkpoint that only saves model weights is a checkpoint that lies about what it does. It will reload, training will resume without an error, and the loss curve will look approximately continuous. But the run after the resume is not the run that was paused. Stochastic draws shift, learning rate jumps, the next batch is wrong, momentum vanishes for one step. Each of those silent divergences corrupts the experiment in a different way.
Bit-exact resume is the discipline of capturing every piece of state the training step depends on, so that a restart from checkpoint produces the same next 100 steps as a continuous run within floating-point noise. The interview question is testing whether you understand the full set, not just the famous one.
The six layers below are listed in the order you should think about them, roughly from heaviest object to smallest counter. The order is also pedagogically useful: each layer answers a different question about what the optimizer step depends on.
Layer 1: model weights
The parameters themselves are the obvious piece, and they are usually the largest single object in the checkpoint. For a 7B model at fp16 that is roughly 14 GB before sharding. Under FSDP or DeepSpeed ZeRO-3 each rank owns a shard, so the on-disk layout is per-rank, but logically the full parameter tensor must be reconstructable.
The failure mode is unique to this layer: without weights you cannot resume at all. There is no silent degradation, just an immediate error. That makes weights the easiest layer to think about and the wrong layer to fixate on. The harder, more interesting layers are the ones whose absence produces a run that looks fine and is silently wrong.
One nuance worth knowing: with mixed-precision training you typically keep an fp32 master copy of the weights for the optimizer to update, then cast to bf16 or fp16 for the forward pass. The master copy is what must be checkpointed; restoring only the low-precision cast loses the rounding tail that the optimizer was working with.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepSpeed's `save_checkpoint` / `load_checkpoint` bundles weights, optimizer, scheduler, RNG, and ZeRO sharding metadata so a preempted run on Llama 4 training can resume bit-exactly across thousands of ranks.
- Hugging Face Accelerate's `save_state` / `load_state` covers all six layers including DataLoader iterator state when the sampler is wrapped in `accelerate.DataLoaderShard`.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does optimizer state often double the checkpoint size of the model weights alone?
Adam stores two moments per parameter at the same dtype as the params, so 2 plus 1 equals 3x param memory at fp32, or 2x once you account for mixed-precision master copies.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saving only model weights and the optimizer, then assuming the resumed run will match the original. Without RNG and data-iterator state the next batch and the next dropout mask both diverge.
60 second bullets to scan on the way to the call.
The six layers of resume state, in order from heaviest object to smallest counter
Why optimizer state is independent of model weights and what Adam stores
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.