Which of these is saved when a fine-tuning checkpoint is written?
A checkpoint saves weights plus Adam moments, scheduler state, RNG state, and step count, everything needed for a near bit-exact resume.
Picture a long road trip you have to pause mid-route. To resume cleanly tomorrow you need more than just where the car ended up. You also need the odometer reading, the next song queued on the playlist, the route plan you were following, and which exit was coming up. Pack only the car's GPS coordinates and you have to redo all the planning when you resume. A training checkpoint works the same way. The weights are where the car is. The optimizer's running averages are the route plan, the scheduler is the playlist, the random seed is the next song, and the step counter is the odometer. Save all of them and you pick up exactly where you left off, with no jarring restart.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A common checkpoint mistake is to save a weights-only file. A hardware crash then kills the job. The weights are restored with fresh optimizer state, and the loss spikes and takes hundreds of steps to recover. The cause is that the optimizer was carrying state every bit as important as the weights themselves.
A correct checkpoint is the serialised version of every mutable piece of the training process that would otherwise be lost on interrupt. That definition forces a precise inventory: weights, optimizer state, scheduler state, RNG state, and step counter at a minimum. Add mixed-precision scaler state if applicable, and sharded parameter and optimizer slices if running under ZeRO or FSDP.
Why optimizer state is the largest and most important component
Adam keeps two running statistics per parameter. The first-moment estimate m is an exponential moving average of past gradients. The second-moment estimate v is an exponential moving average of past gradient squares. Both are stored in fp32 for numerical stability regardless of the training precision, costing 8 bytes per trainable parameter combined.
The update rule uses these moments to scale and normalise each gradient step. The scaling depends on the full history of gradients the run has seen, encoded in the slow decay constants beta1 and beta2 (typically 0.9 and 0.999). Reset the moments to zero on resume and Adam has to rebuild that history from scratch. That takes roughly 1/(1-beta) steps to reach a steady state. For beta2 = 0.999 that is on the order of a thousand steps of distorted learning rate.
The visible signature of this failure is a sharp loss spike at the resume point. It decays over the next several hundred steps as the moments catch up. People sometimes mistake this for the model having lost capability during the interruption. In fact the weights are unchanged. The optimizer just briefly took bad steps.
This is also why optimizer state is the dominant size cost in a checkpoint. For a 7B-parameter full-FT checkpoint in bf16, the weights are 14 GB and the Adam moments are 56 GB. 8-bit Adam cuts the moments to 2 bytes per parameter (14 GB for 7B), the same as the bf16 weights. Factored optimizers such as Adafactor store even less.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Hugging Face Trainer saves a checkpoint directory with model, optimizer.pt, scheduler.pt, rng_state.pth, and trainer_state.json, capturing all five required pieces.
- PyTorch FSDP and DeepSpeed ZeRO use sharded checkpoint formats that store rank-local slices of optimizer state alongside the weights for large-model training.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does Adam show a visible loss spike when its moments are reset to zero on resume?
Trace the update rule. With fresh moments and bias correction, m-hat over sqrt(v-hat) is about ±1 at the first step. So every coordinate moves by about the full LR. This is a sign-like step that ignores the per-parameter scaling the old moments had learned. The estimates stay noisy until the moving averages fill in again. And the LR at the resume point is usually far above the warmup value the run first started with.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saving only the model weights and expecting a clean resume. Adam's moments are lost forever and the optimizer starts fresh, producing a visible loss spike at the resume point.
60 second bullets to scan on the way to the call.
What each of the five standard checkpoint components stores
Why Adam's moments cannot be recomputed from weights and config alone
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.