Adam stores two fp32 buffers per trainable parameter: what does that imply for LoRA vs full FT on a 7B?

Short answer·Medium·4.0 · 0·~3 min·Relevant atDatabricks Meta OpenAI

Attempt it

Adam holds two fp32 buffers (first moment m, second moment v) per trainable parameter. Use that single fact to compute the optimizer-state memory for a full fine-tune of a 7B model versus a LoRA fine-tune at r=16 over a few hundred million trainable parameters. State the headline number for each case and explain why most of LoRA's memory win lives here.

Free · 2 AI evals / day

**Adam** maintains two running averages for every parameter it updates. The first moment m tracks the mean of past gradients, the second moment v tracks their squared magnitude. Both buffers are kept in fp32, which is four bytes each, so the total optimizer cost is eight bytes per trainable parameter. Apply that to a 7-billion-parameter model under full fine-tuning. Every weight is trainable, so 7 billion times eight bytes works out to about 56 GB of optimizer state alone. The model weights themselves in bf16 are another 14 GB, and activations add more on top. The optimizer state is the single biggest line item. Now apply it to **LoRA** at rank 16. Only the small adapter matrices on top of the frozen base are trainable. Depending on which projections you wrap, the trainable count usually lands between 20 and 100 million parameters. At 100 million, the optimizer state is 100 million times eight bytes, about 0.8 GB. That is roughly 70 times less than full fine-tuning. The key idea: LoRA's memory win comes from what you did NOT keep, not from what you did. The base weights still sit in VRAM and activations still flow through the network. What you eliminated is the per-parameter Adam bookkeeping for the billions of frozen weights.

**Eight bytes per trainable parameter** is the load-bearing number. Adam stores two fp32 buffers, the first moment m and the second moment v, for every parameter it updates. Each buffer is four bytes, so the total is eight bytes per trainable parameter, independent of model architecture. Full fine-tune of a 7B model: 7 billion trainable parameters times 8 bytes equals about 56 GB of optimizer state. That is before counting the bf16 weights at 14 GB and the activations, which add another 20 to 40 GB depending on sequence length and batch size. The optimizer state alone is already the largest memory line and the reason a single H100 cannot host a 7B full fine-tune without aggressive techniques like ZeRO-3 or FSDP. LoRA at rank 16 on the same 7B: the trainable count drops to roughly 20 to 100 million parameters depending on which projections are wrapped. At 100 million, optimizer state is 100 million times 8 bytes, about 0.8 GB. That is a roughly 70-fold reduction in the dominant memory line. The reason this is the bulk of LoRA's memory win: optimizer state scales with **trainable** parameter count, not total parameter count. Base weights still sit in VRAM whether you train them or not (in NF4 for QLoRA, bf16 otherwise). Activations still flow through every layer on the forward pass and are still needed for backprop. What LoRA eliminates is the per-parameter Adam bookkeeping for the billions of frozen weights, and that elimination is what fits a 7B fine-tune on a single 24 GB consumer GPU.

Quick reference

TL;DR

Adam costs 8 bytes per trainable parameter; full FT of a 7B model needs about 56 GB of optimizer state, LoRA r=16 needs about 0.8 GB, and that gap is the bulk of the LoRA win.

Memory aid

Easy to grasp

Imagine every worker on a construction site needs two clipboards to track their work. If a thousand workers are on the job, that is two thousand clipboards. If you instead let a small specialist crew of ten people make the actual changes while the rest of the site just observes, you only need twenty clipboards. The clipboards are the bookkeeping the optimiser keeps for each parameter it is changing. Full fine-tuning changes every parameter and pays the cost for all of them. LoRA changes only a small specialist crew of tiny adapter matrices on top of a frozen base model. The base parameters do not move, so they need no clipboards. The big memory saving is the bookkeeping you skipped, not the weights you stored.

Key concepts

Adam optimizer LoRA Memory math for training Activation memory QLoRA

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: compute 8 bytes per trainable parameter, derive 56 GB for full FT vs 0.8 GB for LoRA r=16, and explain that the saving scales with trainable count not total count.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face PEFT's LoRA implementation reports trainable-parameter counts and corresponding VRAM savings of 60-70x on the optimizer line at r=16.
QLoRA paper (Dettmers et al. 2023) showed a 65B fine-tune fitting on a single 48 GB GPU by combining LoRA's optimizer savings with NF4-quantized base weights.
bitsandbytes' Adam8bit drops m and v from fp32 to 8-bit blockwise, cutting optimizer state by a further 4x and used in most consumer-GPU fine-tuning recipes.
Modern fine-tuning stacks like Unsloth and Axolotl default to LoRA with PEFT for 7B to 70B models precisely because the Adam-state win unlocks single-GPU training.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does an 8-bit optimizer like bitsandbytes Adam8bit change the memory math?

QWhat happens to the memory math when you combine LoRA with ZeRO-3 sharding?

QWhy does QLoRA always pair LoRA with 4-bit base weights rather than bf16?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Claiming LoRA saves memory because the adapters are small. The dominant saving is the optimizer state you no longer maintain for the frozen base, not the size of the adapter weights themselves.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.