How does QLoRA fit 65B fine-tuning onto a single 48GB GPU?
List the memory accounting for QLoRA fine-tuning of a 65B model on a single 48 GB GPU. Where does each of QLoRA's three tricks contribute, and what does the LoRA adapter contribute?
65B bf16 is ~130 GB. NF4 + double-quant compresses the frozen base to ~33.5 GB. Adapters stay bf16 (~1.6 GB) with Adam state on adapters only (~6.4 GB). Paged optimisers handle OOM spikes. Total ~45 GB fits.
Imagine you're moving a massive 130-pound encyclopedia into a tiny dorm room that only has a 48-pound shelf. Impossible, until you do four things. First, you photocopy the bulk of the book onto thin rice paper (the base book now weighs about 37 pounds). Second, you shrink the page-margin labels too, knocking off about 3 more pounds. Third, instead of editing the giant book, you stick a small notebook of margin notes (these stay at full quality because you write in them). Fourth, you stash a back-room cabinet so when the desk gets crowded mid-study, papers shuffle out and back. The result fits the shelf. That's the QLoRA stack for squeezing a 65B model into a single 48GB GPU.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Fitting a 65B model onto a single 48GB card is a memory accounting story, not a magic trick. Start from the raw bf16 weight bill, then knock it down one pool at a time. Name which QLoRA trick attacks which pool. Done carefully, the arithmetic lands just under the card's ceiling, and you can see exactly where the slack is.
Everything follows from one split between frozen storage and the learnable path. The base model is frozen, so it can be crushed to 4-bit and carries no optimiser state. The LoRA adapters are learnable, so they stay in bf16 and own the only optimiser state in the run. Two more tricks tidy the edges: double quantisation shaves the quantisation metadata, and paged optimisers stand by as a safety valve for transient spikes.
The starting bill, and the pool that dominates it
Begin with the raw weights. A 65B model in bf16 stores two bytes per parameter, so the weights alone are about 130 GB. That is already nearly three times a 48GB card, before a single byte of training state.
Now add what a full fine-tune would demand on top. Adam keeps two moment estimates per trainable parameter, and gradients add another copy. In fp32 the two Adam moments are eight bytes per parameter, which on 65B is roughly 520 GB by themselves. Stack weights, gradients, and moments together and a full fine-tune wants well over 600 GB across many GPUs.
This pool dominates. The structural idea of LoRA is to delete it. Freeze the base, train only small low-rank adapters, and Adam state stops scaling with the full parameter count. At rank 64 on all linear layers the adapters are about 800M parameters, so their Adam state is roughly 6.4 GB rather than 520 GB. That ~514 GB deletion is the single largest saving in the whole recipe, and it is purely structural. The bit-width tricks below are layered on top of it, not a substitute for it.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Component | Full FT (65B) | LoRA (65B, bf16) | QLoRA (65B, NF4) |
|---|---|---|---|
| Base weights | 130 GB (bf16) | 130 GB (bf16) | ~33.5 GB (NF4 + DQ) |
| Adapter weights | None (no adapters) | ~1.6 GB (+1.6 GB gradients) | ~1.6 GB (+1.6 GB gradients) |
| Adam state | ~520 GB (on base) | ~6.4 GB (on adapter) | ~6.4 GB (on adapter) |
| Activations | ~2 GB (checkpointed, 512 tokens) | ~2 GB (checkpointed, 512 tokens) | ~2 GB (checkpointed, 512 tokens) |
| Total | ~780+ GB with 130 GB of gradients (multi-GPU) | ~142 GB (multi-GPU) | ~45 GB (single 48 GB) |
Real products, models, and research that use this idea.
- Dettmers et al. 2023 demonstrated 65B QLoRA on a single 48 GB GPU, with a reported footprint of about 45 GB at batch size 1 and 512 tokens.
- Axolotl and Unsloth ship 70B QLoRA configs that fit on a single 48 GB or 80 GB GPU.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat's the activation-memory math more precisely at 65B + seq_len 4096?
Each layer's saved input is roughly b × s × d × bytes. At b=1, s=4096, d=8192 in bf16, that's ~64 MB per layer × 80 layers = ~5.4 GB. Checkpointing every layer keeps only those inputs, plus one layer's attention and MLP intermediates during recompute (more without a memory-efficient attention kernel). Without checkpointing, every layer keeps all its intermediates, which runs to tens of GB or more. Added to the ~43 GB of base, adapter and optimizer state, 4096 tokens pushes past 48 GB. The run then needs paged optimizers, a shorter context, or a bigger card.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forgetting that NO Adam state is allocated for the frozen base. Adam state on 65B params in fp32 would be ~520 GB by itself.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.