For full FT of a 7B model in bf16 with standard Adam: weights take GB, gradients take GB, and Adam optimizer states (m and v in fp32) take GB: for a baseline of roughly GB before activations.
Full fine-tuning means updating every weight in the model, so you have to store far more than just the model itself.
For a 7B model in **bf16**, each parameter is 2 bytes. So the weights are 7 billion times 2 bytes, which is 14 GB. The gradients are the same shape as the weights and also stored at 2 bytes each, so they add another 14 GB.
The surprising part is the optimizer. Adam keeps two running averages per parameter, the first moment and the second moment. Both are stored in full 32-bit precision, which is 4 bytes each, so 8 bytes per parameter. That is 7 billion times 8, which is 56 GB.
Add them up: 14 plus 14 plus 56 gives about 84 GB. And that is before you count activations, which depend on batch size and sequence length.
The takeaway: optimizer states, not the weights, dominate the memory bill. This is exactly why people reach for LoRA or QLoRA, which avoid storing full-size states for every parameter.
Full fine-tuning stores four things per parameter: the weight, its gradient, and Adam's two moments. The bytes per parameter cost of each is what drives the total.
In **bf16**, weights are 2 bytes each, so a 7B model holds 14 GB of weights. Gradients mirror the weights at 2 bytes each, adding another 14 GB. Adam keeps the first and second moments in fp32 at 4 bytes each, so 8 bytes per parameter, which is 56 GB.
The total baseline is 14 plus 14 plus 56, roughly 84 GB. Activations sit on top of that and scale with batch size and sequence length.
The key insight is that optimizer states dominate. They are four times the weight footprint, because two fp32 moments cost more than the half-precision weights and gradients combined. An 84 GB baseline does not fit on a single 80 GB H100, so full FT of even a 7B model already needs sharding or multiple GPUs. This is the arithmetic that motivates LoRA, QLoRA, and 8-bit optimizers.
Quick reference
TL;DR
Weights 14 GB, gradients 14 GB, Adam states 56 GB: about 84 GB baseline before activations, which is why full FT of a 7B model needs multiple data-center GPUs.
Picture each of the 7 billion knobs in the model. To train it with Adam, you keep four copies of every knob. One copy is the knob itself, one is the nudge you want to apply this step, and two are running averages Adam keeps to smooth the nudges. The knob and the nudge are stored compactly at 2 bytes each, but the two averages are stored at full 4-byte precision so the math stays stable. Add it up across all 7 billion knobs and you get roughly 84 gigabytes of memory, before you even feed in any data. That is why a model that sounds small still does not fit on one gaming GPU when you train it the full way.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Fitting full fine-tuning of a 7B model onto a single 24 GB consumer GPU The 84 GB baseline alone is over three times that VRAM, so full FT is impossible there. Use LoRA or QLoRA instead.
Estimating peak memory for a long-context or large-batch run This baseline ignores activations, which can exceed the static cost. Add an activation estimate before sizing hardware.
Planning a recipe that keeps an fp32 master copy of the weights That variant adds 4 more bytes per parameter, so the 84 GB figure undercounts the real footprint by 28 GB.
Sizing memory for an 8-bit or paged optimizer setup The 56 GB Adam term assumes fp32 moments. An 8-bit optimizer roughly halves it, breaking this arithmetic.
Quick scan · 2–4 min
Everything important, quickly.
5 min: assign bytes per parameter to weights, grads, and the two Adam moments, multiply by 7e9, sum to 84 GB, then discuss why optimizer states dominate and how LoRA, 8-bit Adam, and ZeRO attack them.
Side-by-side
Component
Bytes/param
7B footprint
Weights (bf16)
2
14 GB
Gradients (bf16)
2
14 GB
Adam moments (2 x fp32)
8
56 GB
Baseline before activations
12
~84 GB
Real-world examples
Real products, models, and research that use this idea.
1Training a Llama 3.1 8B with full FT and standard Adam overflows a single 80 GB H100, which is why teams reach for ZeRO sharding or LoRA in practice.
2DeepSpeed ZeRO-2 shards the 56 GB of Adam states across data-parallel ranks, the exact term this arithmetic shows dominates the budget.
3The bitsandbytes 8-bit Adam cuts moment storage roughly in half, attacking the 56 GB optimizer term directly rather than the weights.
4QLoRA quantizes the frozen base to 4-bit and trains only LoRA adapters, so the 56 GB full-Adam state term collapses to megabytes of adapter moments.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy are Adam's moments kept in fp32 even when the weights are bf16?
Sign in or go Pro to see the approach for this follow-up.
QHow does adding an fp32 master copy of the weights change the 84 GB number?
Sign in or go Pro to see the approach for this follow-up.
QWhere does activation memory sit relative to this 84 GB static baseline?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Counting only the weights and forgetting that Adam keeps two extra fp32 moments per parameter. Those optimizer states are the largest single term, four times the weight footprint.