Which statements about QLoRA are true?
QLoRA stacks four tricks: a 4-bit NF4 base, double quantisation of the constants, paged optimisers, and LoRA adapters kept in bf16.
Imagine shrinking a huge reference encyclopedia so it fits in your backpack. NF4 is a clever compression that keeps the most-used pages crispest, because most pages cluster around the middle. Double quantisation then compresses the little index cards that say how each section was shrunk. The encyclopedia itself you never write in; you keep frozen, compressed pages. Instead you carry a thin notebook of full-quality sticky notes (the adapters) where all your new learning happens. And if your desk runs out of room while studying, paged optimisers quietly move some scratch paper to a drawer and fetch it back later. Together these four tricks let you fine-tune a model that would normally need a server rack on a single gaming GPU.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: the four QLoRA components + why NF4 beats INT4 + what double quant compresses + why adapters stay bf16 + why gradient checkpointing is orthogonal.
| QLoRA component | What it touches | Precision / effect |
|---|---|---|
| NF4 base | Frozen base weights | 4-bit, Normal-distributed buckets |
| Double quantisation | Per-block scaling constants | Saves about 0.37 bits per parameter |
| Paged optimisers | Adam optimiser state | Spills to CPU on memory spikes |
| LoRA adapters | Learnable low-rank factors | Stay in bf16 for stable gradients |
| Gradient checkpointing | Activations (orthogonal) | Still used; saves activation memory |
Real products, models, and research that use this idea.
- Hugging Face PEFT plus bitsandbytes ships QLoRA as the default 4-bit recipe used to fine-tune Llama 3.1 70B on a single 48GB GPU.
- Unsloth and Axolotl both expose NF4 plus double quantisation as one-flag presets for QLoRA runs on consumer and rented GPUs.
- Open-weight tunes of DeepSeek V4 and Qwen3 distributed on the Hugging Face Hub are routinely produced with QLoRA on rented H100 instances.
- Together.ai and Modal Labs reference stacks rely on paged optimisers to keep long QLoRA jobs alive through transient memory spikes.
- Hugging Face TRL's SFTTrainer pairs QLoRA with gradient checkpointing as the standard memory-budget recipe for 2026 instruction tuning.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does NF4 use Normal-distributed quantile buckets instead of uniform spacing?
QWhat numerical problem would arise if you tried to keep the LoRA adapters in 4-bit too?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Thinking the LoRA adapters are also 4-bit, or that a 4-bit base removes the need for gradient checkpointing. Both confuse weight memory with where learning and activations actually live.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.