Adam optimizer state per parameter: fill in the moment buffers and bytes
Two fp32 buffers per parameter: m for the first moment and v for the second, four bytes each, eight bytes of optimizer state per trainable parameter.
Imagine a coach watching every player on a roster across many games. For each player, the coach keeps two notepads. The first notepad records the running average of how much the player moved in each game, in which direction. The second notepad records the running average of how big those moves were, ignoring direction. Both notepads stay open between games so the coach always has a smoothed picture per player. Adam works the same way for parameters. There is one notepad per parameter for the first moment of the correction signal and one for the second moment of that same signal. Each notepad holds a number that takes four bytes of memory. Two notepads at four bytes each adds up to eight bytes of optimizer bookkeeping per parameter.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Adam's memory cost is one of those numbers every fine-tuning engineer should be able to recite from memory. Eight bytes per trainable parameter. That single figure determines whether a 7B fine-tune fits on one 80 GB GPU, whether a 13B fits on two GPUs without sharding, and whether a 70B requires ZeRO-3 or LoRA-style techniques to be trainable at all. Knowing where the number comes from also explains why a half-dozen widely-used techniques exist to reduce it.
The number comes from Adam's design. The optimizer keeps two running averages per parameter to compute adaptive per-parameter learning rates. The first moment m is an exponential moving average of the gradient, capturing its recent direction. The second moment v is an exponential moving average of the squared gradient, capturing its recent magnitude. Both are stored in fp32 even in mixed-precision training, because the running averages accumulate small values over many steps and need the dynamic range that bf16 and fp16 cannot provide.
Two fp32 buffers per parameter at 4 bytes per fp32 scalar gives 8 bytes per parameter for the optimizer state alone. Add the base weights, the gradients, the activations, and any optional fp32 master weight copy and the full training memory picture takes shape. This deep dive walks through what each buffer is for, why fp32 is the right precision, the full memory accounting at common model sizes, and the family of techniques that exist precisely to reduce the 8-byte number.
What m and v actually store
Adam's update at step t for parameter theta uses gradient g_t and looks like:
The parameter update is then:
The hats denote bias-corrected versions that compensate for the initialisation of m and v at zero. The decay rates beta1 around 0.9 and beta2 around 0.999 are conventional and rarely changed.
What m gives you is momentum: a smoothed direction estimate that smooths out gradient noise. What v gives you is per-parameter adaptivity: a smoothed magnitude estimate that scales down updates for parameters with consistently large or noisy gradients and effectively scales up updates for parameters with small consistent gradients.
The combination is what makes Adam work well on the optimisation landscapes that transformers produce. Different parameters have wildly different gradient scales, and per-parameter adaptive scaling is exactly the mechanism that lets one global learning rate do reasonable work across all of them. Plain SGD with a single global step size cannot match this without per-layer learning rate tuning.
Two per-parameter buffers is the minimum the algorithm needs to function. Drop the second moment and you have momentum SGD, which lacks the adaptivity. Drop the first moment and you have a variant of RMSProp, which lacks the momentum smoothing. Both are widely used in specific niches but neither matches Adam on the standard LLM fine-tuning recipe.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Hugging Face Transformers and TRL default to AdamW in fp32 state, which is why their out of the box 7B fine-tune needs roughly 56 GB for optimizer state alone.
- bitsandbytes ships 8-bit AdamW that quantises m and v from fp32 to int8 with block-wise scaling, cutting optimizer state from 8 bytes to about 2 bytes per parameter.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does Adam state need fp32 when the model runs in bf16?
Both m and v accumulate small gradient values over thousands of steps. In bf16 the mantissa is only 7 bits, so small running averages underflow to zero. fp32 preserves the dynamic range that keeps the EMAs meaningful, which matters most for the second moment near the start of training when squared gradients can be tiny.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forgetting that optimizer state is fp32 even in a mixed-precision run. Adam's moments stay in fp32 for numerical stability while the model itself runs in bf16.
60 second bullets to scan on the way to the call.
Why Adam keeps two buffers per parameter and what each one estimates
Why both buffers are fp32 even in a mixed-precision run
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.