Estimate the peak VRAM required to do FULL fine-tuning of a 7B parameter model in bf16 with Adam (no LoRA, no gradient checkpointing). Break down the contribution of weights, gradients, optimizer states, and activations. Why does LoRA cut this so dramatically?
Free · 2 AI evals / day
Training memory is much larger than the model itself, and the optimizer is usually the reason.
Start with the **weights**. A 7B model in bf16 stores 2 bytes per parameter, so 7 billion params is about 14 GB. That part is easy and most people stop here.
But training also stores a gradient for every parameter, which adds roughly another 14 GB. Then Adam keeps two extra buffers per parameter, a running mean and a running variance, both in fp32 at 4 bytes each. That is 8 bytes per parameter, about 56 GB on its own.
Add the activations saved for the backward pass, often 10 to 30 GB, and the total lands near 100 GB. That does not fit on one 80 GB card, so full fine-tuning needs several GPUs.
LoRA changes the math by freezing the base model. Only a tiny adapter is trainable, so gradients and Adam buffers exist for millions of parameters instead of billions. The 56 GB optimizer cost nearly vanishes.
Peak training memory has four parts: weights, gradients, optimizer state, and activations.
The weights of a 7B model in bf16 cost 2 bytes per parameter, so about 14 GB. Gradients mirror the weights at bf16, adding another 14 GB. The optimizer dominates. **Adam** keeps a first and second moment per parameter in fp32, 8 bytes total, so about 56 GB. Activations saved for backprop add another 10 to 30 GB depending on batch and sequence length.
Summing the fixed terms gives 14 plus 14 plus 56, which is 84 GB before activations. With activations the peak sits near 100 GB. A single 80 GB H100 cannot hold it, so you shard across an 8-way cluster with FSDP or ZeRO-3, or offload to CPU.
LoRA cuts this because gradients and optimizer state only exist for trainable parameters. Freeze the base and train a small adapter, and the 56 GB Adam term collapses to under 1 GB. QLoRA quantizes the frozen base to 4-bit, shrinking the 14 GB weight term to about 4 GB too.
Quick reference
TL;DR
Weights 14 GB, gradients 14 GB, fp32 Adam moments 56 GB, plus activations: roughly 100 GB, so full-FT of a 7B needs multi-GPU. LoRA puts the optimizer state on a tiny adapter only.
Think of training as moving house with movers who never throw anything away. The furniture is the model. For every single item, the movers also keep a sticky note saying which way it just shifted, plus two more notes tracking its average drift and how much it tends to bounce around. Those extra notes are bigger than the furniture itself. For a smallish model the furniture is about 14 truck-loads, but all the notes balloon the total past 80, so one truck cannot carry it. The clever trick is to freeze almost all the furniture in place and only let a few small pieces move. Now you only keep notes for those few moving pieces, so the whole job fits in one small truck.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Fine-tuning on a single consumer or pro card with under 48 GB of VRAM Full-FT of a 7B needs roughly 100 GB across the optimizer and weights, far beyond one such card. Use LoRA or QLoRA instead.
You only need a behavior or style shift, not maximum quality LoRA matches full fine-tuning on most style and format tasks while avoiding the 56 GB Adam state blowup entirely.
Budget rules out renting a multi-GPU node for hours Full fine-tuning requires sharding across many GPUs, which multiplies the hourly cost compared to a single-GPU LoRA run.
You have fewer than a few thousand high-quality training examples Full fine-tuning over-parameterizes a tiny dataset and risks forgetting, so a low-rank adapter is the safer and cheaper choice.
Quick scan · 2–4 min
Everything important, quickly.
4 min: byte per param breakdown for each term + the fp32 Adam moment blowup + master weight copy + total near 100 GB + why multi-GPU + how LoRA and QLoRA collapse the optimizer term.
Side-by-side
Memory term
Bytes per param
7B full-FT
7B LoRA (50M trainable)
Weights (bf16)
2
~14 GB
~14 GB (4 GB at 4-bit QLoRA)
Gradients (bf16)
2
~14 GB
<0.2 GB
Adam moments (fp32 m+v)
8
~56 GB
<0.5 GB
fp32 master weights
4
~28 GB
<0.2 GB
Activations
varies
10-30 GB
10-30 GB
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face Accelerate and DeepSpeed ZeRO-3 shard the 56 GB Adam state across an 8xH100 node so a full 7B fine-tune fits where one GPU cannot.
2QLoRA, introduced in 2023, fine-tunes a 65B model on a single 48 GB GPU by 4-bit quantizing the frozen base and training LoRA adapters in bf16.
3Unsloth and Axolotl ship default configs that pick LoRA plus gradient checkpointing precisely because full-FT Adam state on a Llama 3.1 8B exceeds consumer VRAM.
4PyTorch FSDP with CPU offload lets teams run full fine-tuning of Mistral and Qwen bases by paging optimizer moments to host RAM when GPU memory runs short.
5Together.ai and Modal Labs size their full-FT instances around the roughly 16 to 20 bytes per parameter rule, provisioning multi-GPU nodes for 7B-and-up bases.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does mixed-precision Adam need an fp32 master copy of the weights at all?
Sign in or go Pro to see the approach for this follow-up.
QHow does gradient checkpointing change the activation term, and what does it cost?
Sign in or go Pro to see the approach for this follow-up.
QZeRO-3 shards weights, gradients, and optimizer state across N GPUs. What new cost does that introduce?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Quoting only the weight size (14 GB) and concluding a single 80 GB GPU is enough. The optimizer state, not the weights, is what blows the budget past 100 GB.