Which of these is actually saved when a fine-tuning checkpoint is written?

Attempt it

Quick reference

TL;DR

A checkpoint saves weights plus Adam moments, scheduler state, RNG state, and step count, everything needed for a near bit-exact resume.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a long road trip you have to pause mid-route. To resume cleanly tomorrow you need more than just where the car ended up. You also need the odometer reading, the next song queued on the playlist, the route plan you were following, and which exit was coming up. Pack only the car's GPS coordinates and you have to redo all the planning when you resume. A training checkpoint works the same way. The weights are where the car is, but the optimizer's running averages are the route plan, the scheduler is the playlist, the random seed is the next song, and the step counter is the odometer. Save all of them and you pick up exactly where you left off, with no jarring restart.

Key concepts

Training checkpoint Adam optimizer Resuming training

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: enumerate the five required components, walk through why each matters for clean resume, then cover sharded checkpoints and the scaler-state trap in fp16 training.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face Trainer saves a checkpoint directory with model, optimizer.pt, scheduler.pt, rng_state.pth, and trainer_state.json, capturing all five required pieces.
PyTorch FSDP and DeepSpeed ZeRO use sharded checkpoint formats that store rank-local slices of optimizer state alongside the weights for large-model training.
Llama 4 Maverick pretraining runs at Meta resume from sharded checkpoints with thousands of files, one per rank, coordinated by a manifest.
Hugging Face Accelerate exposes save_state and load_state that wrap PyTorch's distributed checkpoint API to capture model, optimizer, scheduler, and RNG state across all ranks.
Anthropic's training infrastructure for Claude Opus 4.7 uses asynchronous checkpoint writes to overlap save IO with compute, hiding the multi-GB optimizer dump behind the next forward pass.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does Adam show a visible loss spike when its moments are reset to zero on resume?

Sign in or go Pro to see the approach for this follow-up.

QHow would you change the checkpointing strategy for a 70B-parameter training run on 128 GPUs?

Sign in or go Pro to see the approach for this follow-up.

QWhat goes wrong if you save the scheduler state but the trainer uses a different warmup length on resume?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Saving only the model weights and expecting a clean resume. Adam's moments are lost forever and the optimizer starts fresh, producing a visible loss spike at the resume point.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.