Estimate the peak VRAM required to do FULL fine-tuning of a 7B parameter model in bf16 with Adam (no LoRA, no gradient checkpointing). Break down the contribution of weights, gradients, optimizer states, and activations. Why does LoRA cut this so dramatically?
Weights 14 GB, gradients 14 GB, fp32 Adam moments 56 GB, plus activations: roughly 100 GB, so full-FT of a 7B needs multi-GPU. LoRA puts the optimizer state on a tiny adapter only.
Think of training as moving house with movers who never throw anything away. The furniture is the model. For every single item, the movers also keep a sticky note saying which way it just shifted, plus two more notes tracking its average drift and how much it tends to bounce around. Those extra notes are bigger than the furniture itself. For a smallish model the furniture is about 14 truck-loads, but all the notes balloon the total past 80, so one truck cannot carry it. The clever trick is to freeze almost all the furniture in place and only let a few small pieces move. Now you only keep notes for those few moving pieces, so the whole job fits in one small truck.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The fastest way to look junior on a training-infrastructure question is to answer the VRAM estimate with the weight size alone. A 7B model in bf16 is about 14 GB, and a single 80 GB H100 has plenty of room for 14 GB. So the naive conclusion is that one GPU suffices. That conclusion is wrong, and the gap between 14 GB and the real footprint is the entire point of the question.
Full fine-tuning stores far more than the weights. For every parameter you train, the system also holds a gradient, two Adam moment buffers, and usually an fp32 master copy of the weight. Stack those per-parameter costs and the optimizer state, not the model, becomes the dominant consumer of memory. The rule of thumb that falls out is roughly 16 to 20 bytes per parameter for full fine-tuning, against 2 bytes for inference.
This deep dive does the arithmetic term by term, sums it to the familiar near-100 GB figure, explains why that forces multi-GPU sharding, and then shows precisely which terms LoRA and QLoRA attack to put the same base model back onto a single card.
Weights and gradients: the 2-byte terms
Start with the model itself. In bf16, each parameter occupies 2 bytes. A 7B model therefore needs 7e9 times 2 bytes, which is 14e9 bytes, or about 14 GB. This is the number everyone quotes, and it is correct, but it is only the first of four terms. Note that bf16 and fp16 are both 16-bit, so both cost 2 bytes per parameter. bf16 is preferred for fine-tuning because its wider exponent range avoids the loss-scaling dance that fp16 demands.
During the backward pass, the autograd engine produces one gradient value per parameter. Gradients are typically kept in bf16 to match the weights, so they cost another 2 bytes per parameter and another 14 GB. Some recipes upcast gradients to fp32 for stability, which would double this term to 28 GB. The gradient buffer is allocated for the duration of the step and freed after the optimizer consumes it, so it counts toward peak memory.
So before the optimizer has stored anything, you are already at 28 GB for a 7B model. That alone is under one 80 GB card, which is exactly why people stop counting too early. The two cheap terms scale with total parameter count, and so do the two expensive terms that follow. The next term is where the budget breaks.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Memory term | Bytes per param | 7B full-FT | 7B LoRA (50M trainable) |
|---|---|---|---|
| Weights (bf16) | 2 | ~14 GB | ~14 GB (4 GB at 4-bit QLoRA) |
| Gradients (bf16) | 2 | ~14 GB | <0.2 GB |
| Adam moments (fp32 m+v) | 8 | ~56 GB | <0.5 GB |
| fp32 master weights | 4 | ~28 GB | <0.2 GB |
| Activations | varies | 10-30 GB | 10-30 GB |
Real products, models, and research that use this idea.
- Hugging Face Accelerate and DeepSpeed ZeRO-3 shard the 56 GB Adam state across an 8xH100 node so a full 7B fine-tune fits where one GPU cannot.
- QLoRA, introduced in 2023, fine-tunes a 65B model on a single 48 GB GPU by 4-bit quantizing the frozen base and training LoRA adapters in bf16.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does mixed-precision Adam need an fp32 master copy of the weights at all?
Reason about precision loss in tiny weight updates. bf16 has too few mantissa bits to accumulate small steps, so the master copy preserves update fidelity while bf16 weights serve the fast forward pass.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Quoting only the weight size (14 GB) and concluding a single 80 GB GPU is enough. The optimizer state, not the weights, is what blows the budget past 100 GB.
60 second bullets to scan on the way to the call.
Byte cost per parameter for weights, gradients, and Adam moments
Why the fp32 moment buffers are the dominant term
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.