Standard Adam keeps fp32 buffers per trainable parameter: the first moment (running mean of gradients) and the second moment (running mean of squared gradients). At 4 bytes each, that is bytes per parameter just for optimizer state, on top of the weights themselves.
**Adam keeps two extra numbers per trainable parameter.** They are called the first moment, written m, and the second moment, written v. They are the running averages that make Adam adaptive.
The first moment m is a smoothed estimate of the recent gradient. It is updated each step as an exponential moving average, controlled by a decay rate like beta1 around 0.9. It behaves like momentum, capturing the direction the parameter has been moving.
The second moment v is a smoothed estimate of the recent squared gradient. It is also an exponential moving average, controlled by beta2 around 0.999. It captures how large the gradients have been, ignoring direction. Dividing the update by the square root of v scales each parameter by how noisy its gradient has been, which is what makes Adam adaptive.
Each buffer holds one fp32 number per trainable parameter. fp32 is 4 bytes. Two buffers times 4 bytes is 8 bytes per parameter, on top of the weights themselves. For a 7B-parameter model, that is about 56 GB of optimizer state, which is why full Adam fine-tuning is so memory-hungry.
**Two fp32 buffers per parameter, named m and v.** Adam maintains an exponential moving average of the gradient as m, and an exponential moving average of the squared gradient as v. They are the first and second moments of the per-parameter gradient distribution.
The update rule looks like:
$$\theta_{t+1} = \theta_t - \text{lr} \cdot \hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon)$$
The hats denote bias-corrected versions. The point of dividing by the square root of v is that parameters whose gradients have been noisy or large get scaled-down updates, while parameters with small consistent gradients get effectively larger steps. That per-parameter adaptivity is what Adam buys you over plain SGD.
The memory cost is concrete. Both m and v are kept in fp32 for numerical stability, even when the model weights and activations are in bf16 or fp16. An fp32 scalar is 4 bytes. Two buffers at 4 bytes each is 8 bytes per trainable parameter, on top of the weights and gradients and activations.
For a 7B model, optimizer state alone is roughly 56 GB. AdamW adds nothing extra: it is Adam plus decoupled weight decay, with the same two buffers. This is the entire reason 8-bit optimizers, paged optimizers, and LoRA exist: they slash this 8-bytes per parameter footprint by quantising, offloading, or shrinking the trainable parameter count.
Quick reference
TL;DR
Two fp32 buffers per parameter: m for the first moment and v for the second, four bytes each, eight bytes of optimizer state per trainable parameter.
Imagine a coach watching every player on a roster across many games. For each player, the coach keeps two notepads. The first notepad records the running average of how much the player moved in each game, in which direction. The second notepad records the running average of how big those moves were, ignoring direction. Both notepads stay open between games so the coach always has a smoothed picture per player. Adam works the same way for parameters. There is one notepad per parameter for the first moment of the correction signal and one for the second moment of that same signal. Each notepad holds a number that takes four bytes of memory. Two notepads at four bytes each adds up to eight bytes of optimizer bookkeeping per parameter.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Memory-constrained fine-tuning where 8 bytes per parameter does not fit Reach for 8-bit AdamW, paged AdamW, or LoRA so the optimizer state cost drops by 4x to 100x; full fp32 Adam is unaffordable at scale on a single GPU.
Inference-only workloads There is no optimizer state at inference; the 8-bytes per parameter accounting only applies during training.
Recipes that explicitly call for SGD or Lion SGD with momentum keeps one buffer (4 bytes); Lion keeps one buffer of momentum and operates differently. The 8-byte Adam number does not apply.
Quick scan · 2–4 min
Everything important, quickly.
4 min: two moments m and v + why both are fp32 + 8-bytes per parameter calculation + total memory implications at 7B and 70B + how 8-bit Adam, paged Adam, ZeRO-1, and LoRA each shrink the number.
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face Transformers and TRL default to AdamW in fp32 state, which is why their out of the box 7B fine-tune needs roughly 56 GB for optimizer state alone.
2bitsandbytes ships 8-bit AdamW that quantises m and v from fp32 to int8 with block-wise scaling, cutting optimizer state from 8 bytes to about 2 bytes per parameter.
3DeepSpeed ZeRO-1 shards the fp32 Adam state across data-parallel ranks, so an N-rank job carries 8/N bytes of optimizer state per parameter per GPU.
4QLoRA recipes pair 4-bit base weights with 8-bit AdamW state on the adapters only, which is what makes 70B-class fine-tuning on a single 80 GB GPU feasible.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does Adam state need fp32 when the model runs in bf16?
Sign in or go Pro to see the approach for this follow-up.
QHow does 8-bit AdamW preserve training quality despite quantising m and v?
Sign in or go Pro to see the approach for this follow-up.
QWhy does ZeRO-1 deliver such a large memory saving in practice?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Forgetting that optimizer state is fp32 even in a mixed-precision run. Adam's moments stay in fp32 for numerical stability while the model itself runs in bf16.