Weights 14 GB, gradients 14 GB, Adam states 56 GB: about 84 GB baseline before activations, which is why full FT of a 7B model needs multiple data-center GPUs.
Picture each of the 7 billion knobs in the model. To train it with Adam, you keep four copies of every knob. One copy is the knob itself, one is the nudge you want to apply this step, and two are running averages Adam keeps to smooth the nudges. The knob and the nudge are stored compactly at 2 bytes each, but the two averages are stored at full 4-byte precision so the math stays stable. Add it up across all 7 billion knobs and you get roughly 84 gigabytes of memory, before you even feed in any data. That is why a model that sounds small still does not fit on one gaming GPU when you train it the full way.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Memory math is a staple of senior fine-tuning interviews because it separates people who have actually trained large models from people who have only read about it. The question looks like trivia, but it tests whether you can decompose training memory into its parts and reason about which part dominates. Interviewers like it because there is a single correct number and a clean derivation, so a candidate either knows the bytes per parameter accounting or does not.
The trap is that the model weights feel like the obvious answer, and they are also the smallest of the three big terms. A 7B model is only 14 GB of weights in half precision, which would fit comfortably on a single GPU. Yet you cannot full-fine-tune it on that same GPU, and the reason is everything you have to store alongside the weights. The gap between the 14 GB intuition and the 84 GB reality is exactly what the question is designed to expose.
The disciplined approach is to assign a fixed bytes per parameter cost to each class of tensor, then multiply by the parameter count exactly once. Doing the multiply once at the end keeps the arithmetic clean and lets you swap precisions or optimizers by changing a single coefficient. This deep dive walks that arithmetic for a 7B model in the standard bf16 Adam recipe, shows why the optimizer states are the dominant term, separates the static baseline from activation memory, and connects the result to the techniques that exist specifically to shrink it.
The four things you store per parameter
Full fine-tuning with Adam keeps four tensors for every trainable parameter. There are the weights themselves, the gradients computed during the backward pass, and the two optimizer state tensors Adam maintains. Every one of these four has exactly as many elements as the model has parameters, so the only thing that varies between them is the bytes per element cost.
The weights and gradients have the same shape as the model. In a bf16 mixed-precision recipe, both are stored at 2 bytes per element. So each of these two terms costs 2 bytes per parameter. The gradient tensor is allocated in full during the backward pass, so you cannot treat it as free or transient when sizing memory.
Adam's two state tensors are the first moment, an exponential moving average of the gradient, and the second moment, an exponential moving average of the squared gradient. Both are kept in fp32 at 4 bytes each. That is 8 bytes per parameter for the optimizer alone, which is the key number people forget. The reason fp32 is used here, even in a bf16 run, is numerical stability. The second moment accumulates squared gradients that are often tiny, and storing them in half precision would let them underflow to zero, which corrupts the per-parameter learning-rate scaling that makes Adam work.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Component | Bytes/param | 7B footprint |
|---|---|---|
| Weights (bf16) | 2 | 14 GB |
| Gradients (bf16) | 2 | 14 GB |
| Adam moments (2 x fp32) | 8 | 56 GB |
| Baseline before activations | 12 | ~84 GB |
Real products, models, and research that use this idea.
- Training a Llama 3.1 8B with full FT and standard Adam overflows a single 80 GB H100, which is why teams reach for ZeRO sharding or LoRA in practice.
- DeepSpeed ZeRO-2 shards the 56 GB of Adam states across data-parallel ranks, the exact term this arithmetic shows dominates the budget.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy are Adam's moments kept in fp32 even when the weights are bf16?
Think about numerical stability of the running variance estimate. The second moment accumulates tiny squared gradients, which underflow in half precision, so fp32 protects the update.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Counting only the weights and forgetting that Adam keeps two extra fp32 moments per parameter. Those optimizer states are the largest single term, four times the weight footprint.
60 second bullets to scan on the way to the call.
Bytes per parameter for bf16 weights and gradients
Why Adam stores two separate fp32 moments
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.