Estimate peak VRAM for a 7B full-FT run in bf16 + Adam
Estimate the peak VRAM required to do FULL fine-tuning of a 7B parameter model in bf16 with Adam (no LoRA, no gradient checkpointing). Break down the contribution of weights, gradients, optimizer states, and activations. Why does LoRA cut this so dramatically?
Weights 14 GB, gradients 14 GB, fp32 Adam moments 56 GB, plus activations: roughly 100 GB, so full-FT of a 7B needs multi-GPU. LoRA puts the optimizer state on a tiny adapter only.
Think of training as moving house with movers who never throw anything away. The furniture is the model. For every single item, the movers also keep a sticky note saying which way it just shifted, plus two more notes tracking its average drift and how much it tends to bounce around. Those extra notes are bigger than the furniture itself. For a smallish model the furniture is about 14 truck-loads, but all the notes balloon the total past 80, so one truck cannot carry it. The clever trick is to freeze almost all the furniture in place and only let a few small pieces move. Now you only keep notes for those few moving pieces, so the whole job fits in one small truck.
Detailed answer & concept explanation~7 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: byte per param breakdown for each term + the fp32 Adam moment blowup + master weight copy + total near 100 GB + why multi-GPU + how LoRA and QLoRA collapse the optimizer term.
| Memory term | Bytes per param | 7B full-FT | 7B LoRA (50M trainable) |
|---|---|---|---|
| Weights (bf16) | 2 | ~14 GB | ~14 GB (4 GB at 4-bit QLoRA) |
| Gradients (bf16) | 2 | ~14 GB | <0.2 GB |
| Adam moments (fp32 m+v) | 8 | ~56 GB | <0.5 GB |
| fp32 master weights | 4 | ~28 GB | <0.2 GB |
| Activations | varies | 10-30 GB | 10-30 GB |
Real products, models, and research that use this idea.
- Hugging Face Accelerate and DeepSpeed ZeRO-3 shard the 56 GB Adam state across an 8xH100 node so a full 7B fine-tune fits where one GPU cannot.
- QLoRA, introduced in 2023, fine-tunes a 65B model on a single 48 GB GPU by 4-bit quantizing the frozen base and training LoRA adapters in bf16.
- Unsloth and Axolotl ship default configs that pick LoRA plus gradient checkpointing precisely because full-FT Adam state on a Llama 3.1 8B exceeds consumer VRAM.
- PyTorch FSDP with CPU offload lets teams run full fine-tuning of Mistral and Qwen bases by paging optimizer moments to host RAM when GPU memory runs short.
- Together.ai and Modal Labs size their full-FT instances around the roughly 16 to 20 bytes per parameter rule, provisioning multi-GPU nodes for 7B-and-up bases.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does mixed-precision Adam need an fp32 master copy of the weights at all?
QHow does gradient checkpointing change the activation term, and what does it cost?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Quoting only the weight size (14 GB) and concluding a single 80 GB GPU is enough. The optimizer state, not the weights, is what blows the budget past 100 GB.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.