Fill in the full-FT memory contributions for a 7B model

Full fine-tuning means updating every weight in the model, so you have to store far more than just the model itself. For a 7B model in **bf16**, each parameter is 2 bytes. So the weights are 7 billion times 2 bytes, which is 14 GB. The gradients are the same shape as the weights and also stored at 2 bytes each, so they add another 14 GB. The surprising part is the optimizer. Adam keeps two running averages per parameter, the first moment and the second moment. Both are stored in full 32-bit precision, which is 4 bytes each, so 8 bytes per parameter. That is 7 billion times 8, which is 56 GB. Add them up: 14 plus 14 plus 56 gives about 84 GB. And that is before you count activations, which depend on batch size and sequence length. The takeaway: optimizer states, not the weights, dominate the memory bill. This is exactly why people reach for LoRA or QLoRA, which avoid storing full-size states for every parameter.

Full fine-tuning stores four things per parameter: the weight, its gradient, and Adam's two moments. The bytes per parameter cost of each is what drives the total. In **bf16**, weights are 2 bytes each, so a 7B model holds 14 GB of weights. Gradients mirror the weights at 2 bytes each, adding another 14 GB. Adam keeps the first and second moments in fp32 at 4 bytes each, so 8 bytes per parameter, which is 56 GB. The total baseline is 14 plus 14 plus 56, roughly 84 GB. Activations sit on top of that and scale with batch size and sequence length. The key insight is that optimizer states dominate. They are four times the weight footprint, because two fp32 moments cost more than the half-precision weights and gradients combined. An 84 GB baseline does not fit on a single 80 GB H100, so full FT of even a 7B model already needs sharding or multiple GPUs. This is the arithmetic that motivates LoRA, QLoRA, and 8-bit optimizers.