Adam optimizer state per parameter: fill in the moment buffers and bytes

**Adam keeps two extra numbers per trainable parameter.** They are called the first moment, written m, and the second moment, written v. They are the running averages that make Adam adaptive. The first moment m is a smoothed estimate of the recent gradient. It is updated each step as an exponential moving average, controlled by a decay rate like beta1 around 0.9. It behaves like momentum, capturing the direction the parameter has been moving. The second moment v is a smoothed estimate of the recent squared gradient. It is also an exponential moving average, controlled by beta2 around 0.999. It captures how large the gradients have been, ignoring direction. Dividing the update by the square root of v scales each parameter by how noisy its gradient has been, which is what makes Adam adaptive. Each buffer holds one fp32 number per trainable parameter. fp32 is 4 bytes. Two buffers times 4 bytes is 8 bytes per parameter, on top of the weights themselves. For a 7B-parameter model, that is about 56 GB of optimizer state, which is why full Adam fine-tuning is so memory-hungry.

**Two fp32 buffers per parameter, named m and v.** Adam maintains an exponential moving average of the gradient as m, and an exponential moving average of the squared gradient as v. They are the first and second moments of the per-parameter gradient distribution. The update rule looks like: $$\theta_{t+1} = \theta_t - \text{lr} \cdot \hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon)$$ The hats denote bias-corrected versions. The point of dividing by the square root of v is that parameters whose gradients have been noisy or large get scaled-down updates, while parameters with small consistent gradients get effectively larger steps. That per-parameter adaptivity is what Adam buys you over plain SGD. The memory cost is concrete. Both m and v are kept in fp32 for numerical stability, even when the model weights and activations are in bf16 or fp16. An fp32 scalar is 4 bytes. Two buffers at 4 bytes each is 8 bytes per trainable parameter, on top of the weights and gradients and activations. For a 7B model, optimizer state alone is roughly 56 GB. AdamW adds nothing extra: it is Adam plus decoupled weight decay, with the same two buffers. This is the entire reason 8-bit optimizers, paged optimizers, and LoRA exist: they slash this 8-bytes per parameter footprint by quantising, offloading, or shrinking the trainable parameter count.