Weights 14 GB, gradients 14 GB, Adam states 56 GB: about 84 GB baseline before activations, which is why full FT of a 7B model needs multiple data-center GPUs.
Picture each of the 7 billion knobs in the model. To train it with Adam, you keep four copies of every knob. One copy is the knob itself, one is the nudge you want to apply this step, and two are running averages Adam keeps to smooth the nudges. The knob and the nudge are stored compactly at 2 bytes each. The two averages are stored at full 4-byte precision so the math stays stable. Add it up across all 7 billion knobs and you get roughly 84 gigabytes of memory, before you even feed in any data. That is why a model that sounds small still does not fit on one gaming GPU when you train it the full way.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Training memory decomposes into a few parts, and one of them dominates.
The model weights feel like the obvious answer, but they tie with the gradients as the smallest of the three big terms. A 7B model is only 14 GB of weights in half precision, which would fit comfortably on a single GPU. Yet you cannot full-fine-tune it on that same GPU, and the reason is everything you have to store alongside the weights. The real baseline is 84 GB, not the 14 GB intuition.
The disciplined approach is to assign a fixed bytes per parameter cost to each class of tensor, then multiply by the parameter count exactly once. Doing the multiply once at the end keeps the arithmetic clean and lets you swap precisions or optimizers by changing a single coefficient.
The four things you store per parameter
Full fine-tuning with Adam keeps four tensors for every trainable parameter. There are the weights themselves, the gradients computed during the backward pass, and the two optimizer state tensors Adam maintains. Every one of these four has exactly as many elements as the model has parameters. The only thing that varies between them is the bytes per element cost.
The weights and gradients have the same shape as the model. In a bf16 mixed-precision recipe, both are stored at 2 bytes per element. So each of these two terms costs 2 bytes per parameter. The backward pass allocates the full gradient tensor, so you cannot treat it as free or transient when sizing memory.
Adam keeps two state tensors. The first moment is an exponential moving average of the gradient. The second moment is an exponential moving average of the squared gradient. Both are kept in fp32 at 4 bytes each. That is 8 bytes per parameter for the optimizer alone. The reason fp32 is used here, even in a bf16 run, is numerical stability. The second moment accumulates squared gradients that are often tiny. In fp16 they would underflow to zero. bf16 has the fp32 range but only 7 mantissa bits, so small changes to the running averages get rounded away. Either way, this corrupts the learning rate scaling for each parameter that makes Adam work.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Component | Bytes/param | 7B footprint |
|---|---|---|
| Weights (bf16) | 2 | 14 GB |
| Gradients (bf16) | 2 | 14 GB |
| Adam moments (2 x fp32) | 8 | 56 GB |
| Baseline before activations | 12 | ~84 GB |
Real products, models, and research that use this idea.
- Training a Llama 3.1 8B with full FT and standard Adam overflows a single 80 GB H100, which is why teams reach for ZeRO sharding or LoRA in practice.
- DeepSpeed ZeRO-2 shards the 56 GB of Adam states across data-parallel ranks, the exact term this arithmetic shows dominates the budget.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy are Adam's moments kept in fp32 even when the weights are bf16?
Think about numerical stability of the running variance estimate. The second moment accumulates tiny squared gradients, which underflow in fp16 and lose precision in bf16, so fp32 protects the update.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Counting only the weights and forgetting that Adam keeps two extra fp32 moments per parameter. Those optimizer states are the largest single term, four times the weight footprint.
60 second bullets to scan on the way to the call.
Bytes per parameter for bf16 weights and gradients
Why Adam stores two separate fp32 moments
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.