Walk through the memory accounting for QLoRA fine-tuning of a 65B model on a single 48 GB GPU. Where does each of QLoRA's three tricks contribute, and what does the LoRA adapter contribute?
Free · 2 AI evals / day
Let's walk the memory math for a 65B QLoRA fine-tune on a 48 GB GPU. A 65B model in bf16 takes 130 GB just for the weights, which is nearly 3× our GPU. Full fine-tuning would add optimizer state and gradients on top, pushing past 600 GB and out of reach on one card.
QLoRA layers four tricks to close that gap. First, NF4 is a 4-bit NormalFloat format tuned for the bell-curve shape of LLM weights, and it compresses the frozen base from 130 GB to about 33 GB. Second, double quantization shrinks the scaling factors that NF4 itself uses, shaving off another ~3 GB. Third, LoRA adds tiny trainable adapter layers in bf16, only ~200 million parameters, and the optimizer state lives only on those adapters, never on the frozen base. Fourth, paged optimizers let that optimizer state spill to CPU memory when GPU usage spikes.
Add it up: ~30 GB base + ~0.4 GB adapter + ~1.6 GB optimizer + ~12 GB activations fits in 48 GB.
65B in bf16 weights alone is 130 GB, and full fine-tune would add another 520 GB of Adam state on top. QLoRA gets that down to roughly 45 GB on a single card via four tricks layered on top of LoRA's structural saving.
Starting point: 65B in bf16 is 130 GB of weights. Full fine-tune adds Adam state at 8 bytes per parameter. That's another 520 GB on top, so we're nowhere near a single 48 GB card.
The first trick is NF4. NormalFloat-4 quantizes the frozen base from 2 bytes per parameter to roughly 0.5 bytes, dropping the base from 130 GB to about 33 GB. It uses quantile buckets matched to the Gaussian shape of LLM weights, so quality loss is small.
The second trick is double quantization. The per-block FP32 scales used by NF4 themselves get quantized, saving roughly 0.4 bits per parameter. Base lands around 30 GB.
The third trick, and the real win, is LoRA on top. Adapters at rank 64 are about 200M parameters in bf16, around 0.4 GB. Adam state lives only on the adapters, so optimizer memory drops from 520 GB to about 1.6 GB. That's the structural saving.
The fourth trick is paged optimizers. When activations peak during backward, optimizer state pages out to CPU over CUDA unified memory, then pages back.
Add checkpointed activations (~12 GB) and headroom, totals around 45-50 GB. The trade-off is dequant overhead at every matmul, but for training that's acceptable.
Quick reference
TL;DR
65B bf16 is ~130 GB. NF4 + double-quant compresses the frozen base to ~30 GB. Adapters stay bf16 (~0.4 GB) with Adam state on adapters only (~1.5 GB). Paged optimisers handle transient OOMs. Total fits in 48 GB.
Imagine you're moving a massive 130-pound encyclopedia into a tiny dorm room that only has a 48-pound shelf. Impossible, until you do four things. First, you photocopy the bulk of the book onto thin rice paper (the base book now weighs only 33 pounds). Second, you shrink the page-margin labels too, knocking off a few more pounds. Third, instead of editing the giant book, you stick a small notebook of margin notes (these stay at full quality because you actually write in them). Fourth, you stash a back-room cabinet so when the desk gets crowded mid-study, papers shuffle out and back. The result fits the shelf. That's the QLoRA stack for squeezing a 65B model into a single small GPU.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Latency-critical inference (<50ms p99) NF4 dequant adds 10-15% per forward pass; at high QPS the bandwidth cost dominates and the memory win stops paying for itself.
Tiny base models (<3B parameters) Memory isn't the bottleneck at this scale, full fine-tune is faster, simpler, and matches quality without dequant overhead.
Learning entirely new domains, not just behavior shift A ~200M-parameter adapter can't carry that much new information; you need full fine-tune or domain pre-training to absorb new distributions.
Hundreds of adapters served on one fleet Adapter explosion turns serving into a routing problem, at that point MoE-LoRA, S-LoRA-style multi-tenant routing, or a unified head beats stacking QLoRA adapters.
Hardware without efficient 4-bit kernels Without bitsandbytes-class kernels, NF4 dequant becomes a serial CPU-side bottleneck and you lose the throughput benefit entirely.
Quick scan · 2–4 min
Everything important, quickly.
Lay out the full table: base weights (NF4 + DQ → ~30 GB), adapters (bf16 → 0.4 GB), Adam state (adapter-only fp32 → 1.6 GB), activations (checkpointed → 10-15 GB), KV cache + overhead (~2-5 GB). Emphasise that the biggest single saving is structural: no Adam state on the frozen 65B, which alone is 520 GB. Bit-width tricks are on top. Paged optimisers are the safety net for transient memory spikes.
Side-by-side
Component
Full FT (65B)
LoRA (65B, bf16)
QLoRA (65B, NF4)
Base weights
130 GB (bf16)
130 GB (bf16)
~30 GB (NF4 + DQ)
Adapter weights
:
~0.4 GB
~0.4 GB
Adam state
~520 GB (on base)
~1.6 GB (on adapter)
~1.6 GB (on adapter)
Activations
~15 GB (checkpointed)
~15 GB (checkpointed)
~12 GB (checkpointed)
Total
~650+ GB (multi-GPU)
~150 GB (multi-GPU)
~45-50 GB (single 48 GB)
Real-world examples
Real products, models, and research that use this idea.
1Dettmers et al. 2023 demonstrated 65B QLoRA on a single 48 GB A6000.
2Axolotl and Unsloth ship 70B QLoRA configs that fit on a single 48 GB or 80 GB GPU.
3Together AI and Predibase offer hosted QLoRA fine-tuning of 70B models on single-GPU instances.
4Hugging Face's `peft` + `bitsandbytes` integration is the canonical implementation, used by thousands of community fine-tunes.
5LLaMA-3-70B QLoRA fine-tunes on Hugging Face Hub are routinely produced on single 48 GB consumer-grade or workstation GPUs.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat's the activation-memory math more precisely at 65B + seq_len 4096?
Sign in or go Pro to see the approach for this follow-up.
QCould you push QLoRA to fit 65B on a 24 GB consumer GPU?
Sign in or go Pro to see the approach for this follow-up.
QWhen do paged optimisers actually cost performance?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Forgetting that NO Adam state is allocated for the frozen base. Adam state on 65B params in fp32 would be ~520 GB by itself.