Adam stores two fp32 buffers per trainable parameter: what does that imply for LoRA vs full FT on a 7B?

Short answer·Medium·4.0 · 0·~3 min·Asked atDescript Ltimindtree Snowflake·Relevant atAnthropic Databricks Meta OpenAI

Attempt it

Adam holds two fp32 buffers (first moment m, second moment v) per trainable parameter. Use that single fact to compute the optimizer state memory for a full fine-tune of a 7B model versus a LoRA fine-tune at r=16 over a few hundred million trainable parameters. State the headline number for each case and explain why most of LoRA's memory win lives here.

Free · 2 AI evals / day

**Adam** maintains two running averages for every parameter it updates. The first moment m tracks the mean of past gradients, the second moment v tracks their squared magnitude. Both buffers are kept in fp32, which is four bytes each, so the total optimizer cost is eight bytes per trainable parameter. Apply that to a 7-billion-parameter model under full fine-tuning. Every weight is trainable, so 7 billion times eight bytes works out to about 56 GB of optimizer state alone. The model weights themselves in bf16 are another 14 GB, and activations add more on top. The optimizer state is the single biggest line item. Now apply it to **LoRA** at rank 16. Only the small adapter matrices on top of the frozen base are trainable. Depending on which projections you wrap, the trainable count usually lands between 20 and 100 million parameters. At 100 million, the optimizer state is 100 million times eight bytes, about 0.8 GB. That is roughly 70 times less than full fine-tuning. The key idea: LoRA's memory win comes from what you did NOT keep, not from what you did. The base weights still sit in VRAM and activations still flow through the network. What you eliminated is the per-parameter Adam bookkeeping for the billions of frozen weights.

**Eight bytes per trainable parameter** is the load-bearing number. Adam stores two fp32 buffers, the first moment m and the second moment v, for every parameter it updates. Each buffer is four bytes, so the total is eight bytes per trainable parameter, independent of model architecture. Full fine-tune of a 7B model: 7 billion trainable parameters times 8 bytes equals about 56 GB of optimizer state. That is before counting the bf16 weights at 14 GB and the activations, which add another 20 to 40 GB depending on sequence length and batch size. The optimizer state alone is already the largest memory line and the reason a single H100 cannot host a 7B full fine-tune without aggressive techniques like ZeRO-3 or FSDP. LoRA at rank 16 on the same 7B: the trainable count drops to roughly 20 to 100 million parameters depending on which projections are wrapped. At 100 million, optimizer state is 100 million times 8 bytes, about 0.8 GB. That is a roughly 70-fold reduction in the dominant memory line. The reason this is the bulk of LoRA's memory win: optimizer state scales with **trainable** parameter count, not total parameter count. Base weights still sit in VRAM whether you train them or not (in NF4 for QLoRA, bf16 otherwise). Activations still flow through every layer on the forward pass and are still needed for backprop. What LoRA eliminates is the per-parameter Adam bookkeeping for the billions of frozen weights, and that elimination is what fits a 7B fine-tune on a single 24 GB consumer GPU.

TL;DR

Adam costs 8 bytes per trainable parameter; full FT of a 7B model needs about 56 GB of optimizer state, LoRA r=16 needs about 0.8 GB, and that gap is the bulk of the LoRA win.

Memory aid

Easy to grasp

Imagine every worker on a construction site needs two clipboards to track their work. If a thousand workers are on the job, that is two thousand clipboards. If you instead let a small specialist crew of ten people make the actual changes while the rest of the site just observes, you only need twenty clipboards. The clipboards are the bookkeeping the optimiser keeps for each parameter it is changing. Full fine-tuning changes every parameter and pays the cost for all of them. LoRA changes only a small specialist crew of tiny adapter matrices on top of a frozen base model. The base parameters do not move, so they need no clipboards. The big memory saving is the bookkeeping you skipped, not the weights you stored.

Key concepts

Adam optimizer LoRA Memory math for training Activation memory QLoRA

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Memory math is the most useful kind of arithmetic in fine-tuning. It tells you in one minute whether a recipe will fit on the hardware you have, and it explains why every popular efficiency trick exists. The numbers are small, the formulas are arithmetic, and the implications are large.

This question pulls on a single thread of that math: the optimizer state cost. Adam, the default optimizer for almost every fine-tuning recipe, stores two running averages per trainable parameter. The first moment m is the exponentially weighted mean of past gradients. The second moment v is the exponentially weighted mean of past squared gradients. Both are kept in fp32 regardless of the precision the weights themselves use, because the update step needs the higher precision to remain numerically stable. Each fp32 buffer is four bytes, so the per-parameter optimizer cost is exactly eight bytes.

That fact, applied at the scale of a 7-billion-parameter model, exposes why full fine-tuning is so expensive and why LoRA is so cheap. Full fine-tuning pays the eight-byte cost for every one of the seven billion parameters. LoRA pays it only for the small adapter matrices on top of a frozen base. The ratio between those two trainable counts is the ratio between the two memory bills, and that ratio is roughly seventy. Everything else about the LoRA value proposition flows from this single line.

The arithmetic for full fine-tuning

A 7-billion-parameter model trained with vanilla Adam has 7 billion trainable parameters by definition. The optimizer state at eight bytes per parameter is 7 billion times 8, which is 56 billion bytes, or about 56 GB. That is the largest single line in the memory budget.

Add the other lines. The weights themselves in bf16 are two bytes per parameter, so 14 GB. The gradient buffer is the same size as the weights and lives during the backward pass, another 14 GB. The activations needed for backprop scale with sequence length and batch size and typically land in the 20 to 40 GB range for a 4096-token sequence at batch size 1. The total at one batch on one GPU exceeds 100 GB.

This is why full fine-tuning a 7B model does not fit on a single 80 GB H100. The standard production answers are sharding the optimizer state across multiple GPUs with ZeRO-2 or full sharding of optimizer, gradient, and weight state with ZeRO-3 or FSDP. Both schemes work, both add network communication overhead, and both require multi-GPU infrastructure that not every team has access to.

Look one step further. For an A100 80 GB cluster, a 7B full fine-tune typically wants four to eight GPUs to hold the model, optimizer, and activations comfortably with reasonable batch sizes. For a 70B model, the same math says 56 GB times ten, plus weights and activations, and the cluster jumps to thirty-two to sixty-four GPUs. The optimizer state is the single line that drives those scale-up decisions.

The arithmetic for LoRA

Why the optimizer saving dominates

Variants of the optimizer-state line

Stacking with QLoRA and ZeRO

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Hugging Face PEFT's LoRA implementation reports trainable-parameter counts and corresponding VRAM savings of 60-70x on the optimizer line at r=16.
QLoRA paper (Dettmers et al. 2023) showed a 65B fine-tune fitting on a single 48 GB GPU by combining LoRA's optimizer savings with NF4-quantized base weights.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does an 8-bit optimizer like bitsandbytes Adam8bit change the memory math?

Adam8bit stores m and v in 8-bit blockwise quantization rather than fp32. That cuts optimizer state from 8 bytes to 2 bytes per trainable parameter, another 4x saving on the line item this question is about.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Claiming LoRA saves memory because the adapters are small. The dominant saving is the optimizer state you no longer maintain for the frozen base, not the size of the adapter weights themselves.

60 second bullets to scan on the way to the call.

Why Adam stores exactly two fp32 buffers per trainable parameter
How to compute optimizer state cost for any model and any trainable count

Primary sources. Browse if you want the original framing.