1Training-step counter and bookkeeping (epoch index, global step, gradient-accumulation micro-step counter)
2Learning-rate scheduler state (so warmup/decay continues from the right step, not step 0)
3Data sampler / DataLoader iterator position (so the very next batch is the one the run would have seen)
4RNG state across Python, NumPy, PyTorch CPU, and CUDA (so dropout, augmentations, and shuffles are reproducible)
5Optimizer state (Adam moments m and v, plus the global step counter)
6Model weights (the parameters themselves)
A training **checkpoint** is much more than the model weights. To resume a run bit-exactly, six different pieces of state all need to come back.
First, the model weights themselves. Without them you have nothing to resume.
Second, the optimizer state. Adam tracks two moving averages per parameter and a step counter. Lose them and the very next update direction is wrong.
Third, the learning-rate scheduler. Warmup and cosine decay depend on knowing how many steps have already passed. A fresh scheduler restarts at step 0 and applies the wrong LR.
Fourth, RNG state across Python, NumPy, PyTorch CPU, and CUDA. Dropout masks, data augmentations, and shuffles all draw from those generators. Fresh seeds produce different masks and the run drifts.
Fifth, the data iterator position. You want the next batch to be the one the original run would have seen, not batch 0 again.
Sixth, training counters: epoch, global step, and the gradient-accumulation micro-step. Logging, eval cadence, and stopping all depend on them.
Bit-exact resume needs six layers of state, and the order matters because each later layer assumes the earlier ones are already in place.
**Model weights** come first; they are the largest object and the resume is meaningless without them. **Optimizer state** comes next: Adam stores moments `m` and `v` per parameter plus the step counter, and a missing optimizer makes the first post-resume update a fresh-Adam update with no momentum.
The **LR scheduler** is third. Warmup and cosine decay are functions of step index, so if the scheduler restarts at zero you re warm-up and apply a learning rate that has nothing to do with where training actually was.
**RNG state** is fourth: Python, NumPy, PyTorch CPU, and CUDA generators each need their own state. Dropout masks, data augmentations, and shuffle orders all draw from them, and a fresh seed silently changes every stochastic decision from step one.
The **data iterator** is fifth. The next batch must be the one the original run would have served next, not example zero. Finally, step counters and bookkeeping, epoch index, global step, and the gradient-accumulation micro-step pointer, keep logging, eval cadence, and the LR scheduler consistent.
Miss any one layer and the run drifts in a different way: drop optimizer state and momentum dies, drop RNG and dropout patterns shift, drop the iterator and you re-train on already-seen batches.
Quick reference
TL;DR
Restore weights, then optimizer state, then LR scheduler, then RNG, then data iterator, then step counters. Each missing layer breaks bit-exact resume in a different way.
Imagine pausing a long board game and coming back tomorrow. The pieces on the board are the model weights. The notes your players wrote about how aggressively each one was playing are the optimizer state. The timer showing how far into the tournament you are is the scheduler. The shuffled deck and the dice memory are the RNG. The exact spot in the playlist of question cards is the data iterator. And the scorecard with round numbers is the bookkeeping. If you reload only the pieces and lose the rest, the game continues but it is not the same game. Bit-exact resume means restoring every single one of those, in the right order, so the next move is identical to what tomorrow would have been.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Short fine-tunes that finish well under the preemption window, where one continuous run is cheaper than building bit-exact resume The engineering cost of capturing every state layer is not justified if you never actually need to resume mid-run.
Exploratory hyperparameter sweeps where each trial starts from a clean checkpoint anyway Bit-exact resume of an aborted sweep trial buys nothing; you would relaunch the trial from the original starting point.
Inference-only deployment of a finished model Serving only needs the weights and tokenizer; optimizer, scheduler, RNG, and iterator state are irrelevant once training is complete.
Stochastic-by-design ensembling where each restart should sample a new dropout trajectory Some training recipes deliberately reseed on resume to encourage exploration; bit-exact replay would defeat the design.
Quick scan · 2–4 min
Everything important, quickly.
5 min: six resume layers in order + per-layer failure mode + RNG generators + DataLoader iterator state + grad-accumulation micro-step + framework support in 2026.
Real-world examples
Real products, models, and research that use this idea.
1DeepSpeed's `save_checkpoint` / `load_checkpoint` bundles weights, optimizer, scheduler, RNG, and ZeRO sharding metadata so a preempted run on Llama 4 training can resume bit-exactly across thousands of ranks.
2Hugging Face Accelerate's `save_state` / `load_state` covers all six layers including DataLoader iterator state when the sampler is wrapped in `accelerate.DataLoaderShard`.
3PyTorch FSDP exposes `FSDP.full_optim_state_dict` so optimizer moments resume correctly under sharded training in 2026 production stacks.
4Mosaic Composer checkpoints include per-rank `numpy` and CUDA RNG snapshots so distributed dropout masks reproduce on resume.
5Anthropic and OpenAI publish post-mortems describing silent training divergence after a preemption when the data iterator was rebuilt from scratch instead of restored from state.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does optimizer state often double the checkpoint size of the model weights alone?
Sign in or go Pro to see the approach for this follow-up.
QHow would you verify a checkpoint actually resumes bit-exactly rather than approximately?
Sign in or go Pro to see the approach for this follow-up.
QWhat changes under FSDP or DeepSpeed ZeRO-3 sharded checkpointing?
Sign in or go Pro to see the approach for this follow-up.
QWhy is gradient-accumulation micro-step state easy to forget and dangerous to skip?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Saving only model weights and the optimizer, then assuming the resumed run will match the original. Without RNG and data-iterator state the next batch and the next dropout mask both diverge.