Which of these is actually saved when a fine-tuning checkpoint is written?
A checkpoint saves weights plus Adam moments, scheduler state, RNG state, and step count, everything needed for a near bit-exact resume.
Picture a long road trip you have to pause mid-route. To resume cleanly tomorrow you need more than just where the car ended up. You also need the odometer reading, the next song queued on the playlist, the route plan you were following, and which exit was coming up. Pack only the car's GPS coordinates and you have to redo all the planning when you resume. A training checkpoint works the same way. The weights are where the car is, but the optimizer's running averages are the route plan, the scheduler is the playlist, the random seed is the next song, and the step counter is the odometer. Save all of them and you pick up exactly where you left off, with no jarring restart.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Most people learn what a checkpoint is by getting it wrong once. They save a weights-only file, lose a job to a hardware crash, restore the weights with fresh optimizer state, watch the loss spike for a thousand steps, and only then realise that the optimizer was carrying state every bit as important as the weights themselves.
A correct checkpoint is the serialised version of every mutable piece of the training process that would otherwise be lost on interrupt. That definition forces a precise inventory: weights, optimizer state, scheduler state, RNG state, and step counter at a minimum, plus mixed-precision scaler state if applicable, plus sharded parameter and optimizer slices if running under ZeRO or FSDP.
This deep dive walks through each component, explains why omitting it breaks resume in a specific observable way, and then covers the distributed-training extensions that are increasingly the default for any serious fine-tuning run.
Why optimizer state is the largest and most important component
Adam keeps two running statistics per parameter: a first-moment estimate m that is an exponential moving average of past gradients, and a second-moment estimate v that is an exponential moving average of past gradient squares. Both are stored in fp32 for numerical stability regardless of the training precision, costing 8 bytes per trainable parameter combined.
The update rule uses these moments to scale and normalise each gradient step. The scaling depends on the full history of gradients the run has seen, encoded in the slow decay constants beta1 and beta2 (typically 0.9 and 0.999). Reset the moments to zero on resume and Adam has to rebuild that history from scratch, taking roughly 1/(1-beta) steps to reach a steady state. For beta2 = 0.999 that is on the order of a thousand steps of distorted learning rate.
The visible signature of this failure is a sharp loss spike at the resume point that decays over the next several hundred steps as the moments catch up. People sometimes mistake this for the model having lost capability during the interruption; in fact the weights are unchanged, the optimizer just briefly took bad steps.
This is also why optimizer state is the dominant size cost in a checkpoint. For a 7B-parameter full-FT checkpoint in bf16, the weights are 14 GB and the Adam moments are 56 GB. Even after recent moves to 8-bit Adam or factored optimizers, the optimizer state remains the heaviest single contributor.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Hugging Face Trainer saves a checkpoint directory with model, optimizer.pt, scheduler.pt, rng_state.pth, and trainer_state.json, capturing all five required pieces.
- PyTorch FSDP and DeepSpeed ZeRO use sharded checkpoint formats that store rank-local slices of optimizer state alongside the weights for large-model training.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does Adam show a visible loss spike when its moments are reset to zero on resume?
Trace the update rule. The first few steps with zero moments produce un-normalised gradient steps and bias-correction kicks in slowly, so the effective learning rate is wrong until the moving averages stabilise.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saving only the model weights and expecting a clean resume. Adam's moments are lost forever and the optimizer starts fresh, producing a visible loss spike at the resume point.
60 second bullets to scan on the way to the call.
What each of the five standard checkpoint components stores
Why Adam's moments cannot be recomputed from weights and config alone
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.