How does QLoRA fit a 65B fine-tune onto a single 48GB GPU?
Walk through the memory accounting for QLoRA fine-tuning of a 65B model on a single 48 GB GPU. Where does each of QLoRA's three tricks contribute, and what does the LoRA adapter contribute?
65B bf16 is ~130 GB. NF4 + double-quant compresses the frozen base to ~30 GB. Adapters stay bf16 (~0.4 GB) with Adam state on adapters only (~1.5 GB). Paged optimisers handle transient OOMs. Total fits in 48 GB.
Imagine you're moving a massive 130-pound encyclopedia into a tiny dorm room that only has a 48-pound shelf. Impossible, until you do four things. First, you photocopy the bulk of the book onto thin rice paper (the base book now weighs only 33 pounds). Second, you shrink the page-margin labels too, knocking off a few more pounds. Third, instead of editing the giant book, you stick a small notebook of margin notes (these stay at full quality because you actually write in them). Fourth, you stash a back-room cabinet so when the desk gets crowded mid-study, papers shuffle out and back. The result fits the shelf. That's the QLoRA stack for squeezing a 65B model into a single small GPU.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Fitting a 65B model onto a single 48GB card is a memory-accounting story, not a magic trick. The right way to answer is to start from the raw bf16 weight bill, then knock it down one pool at a time, naming which QLoRA trick attacks which pool. Done carefully, the arithmetic lands just under the card's ceiling, and you can see exactly where the slack is.
The single mental model that organises everything is a split between frozen storage and the learnable path. The base model is frozen, so it can be crushed to 4-bit and carries no optimiser state. The LoRA adapters are learnable, so they stay in bf16 and own the only optimiser state in the run. Two more tricks tidy the edges: double quantisation shaves the quantisation metadata, and paged optimisers stand by as a safety valve for transient spikes.
This deep dive walks the four pools in the order that matters for the math. It takes the structural optimiser saving first, then bit-width compression of the frozen base, then the adapter and its state, then activations and headroom. A worked example then runs the full ledger end to end. You will see the 48GB number fall out of the parts rather than appear by assertion.
The starting bill, and the pool that dominates it
Begin with the raw weights. A 65B model in bf16 stores two bytes per parameter, so the weights alone are about 130 GB. That is already nearly three times a 48GB card, before a single byte of training state.
Now add what a full fine-tune would demand on top. Adam keeps two moment estimates per trainable parameter, and gradients add another copy. In fp32 the two Adam moments are eight bytes per parameter, which on 65B is roughly 520 GB by themselves. Stack weights, gradients, and moments together and a full fine-tune wants well over 600 GB across many GPUs.
This is the pool that dominates, and the structural insight of LoRA is to delete it. Freeze the base, train only small low-rank adapters, and Adam state stops scaling with the full parameter count. At rank 64 the adapters are on the order of 200M parameters, so their Adam state is roughly 1.6 GB rather than 520 GB. That ~518 GB deletion is the single largest saving in the whole recipe, and it is purely structural. The bit-width tricks below are layered on top of it, not a substitute for it.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Component | Full FT (65B) | LoRA (65B, bf16) | QLoRA (65B, NF4) |
|---|---|---|---|
| Base weights | 130 GB (bf16) | 130 GB (bf16) | ~30 GB (NF4 + DQ) |
| Adapter weights | : | ~0.4 GB | ~0.4 GB |
| Adam state | ~520 GB (on base) | ~1.6 GB (on adapter) | ~1.6 GB (on adapter) |
| Activations | ~15 GB (checkpointed) | ~15 GB (checkpointed) | ~12 GB (checkpointed) |
| Total | ~650+ GB (multi-GPU) | ~150 GB (multi-GPU) | ~45-50 GB (single 48 GB) |
Real products, models, and research that use this idea.
- Dettmers et al. 2023 demonstrated 65B QLoRA on a single 48 GB A6000.
- Axolotl and Unsloth ship 70B QLoRA configs that fit on a single 48 GB or 80 GB GPU.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat's the activation-memory math more precisely at 65B + seq_len 4096?
Per layer, forward activations are roughly b × s × d × bytes plus attention's intermediate scores. At b=1, s=4096, d=8192, that's ~64 MB per layer × 80 layers = 5 GB pure forward without intermediate buffers. With backward storage, double or triple. Gradient checkpointing every 8 layers reduces stored activations to ~10 layers' worth = ~50 GB → still tight. Typical QLoRA configs use even more aggressive checkpointing (every 2-4 layers).
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forgetting that NO Adam state is allocated for the frozen base. Adam state on 65B params in fp32 would be ~520 GB by itself.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.