**QLoRA** is a recipe for fine-tuning a very large model on a single GPU. It bolts four memory tricks together, and the question is testing whether you know which trick does what.
The frozen base model is stored in 4-bit using a format called NF4. NF4 buckets are spaced to match how weights are distributed, clustered near zero, so it loses less than plain 4-bit integers. That is option A, and it is true.
Double quantisation then compresses the tiny scaling constants that NF4 needs, squeezing out roughly another 0.4 bits per parameter. That is option B, also true. Paged optimisers spill optimiser state to CPU when the GPU is about to run out of memory, then bring it back. That is option D, true.
The two traps are easy to fall for. The LoRA adapters are NOT 4-bit; they stay in bf16 so gradients are stable, so option C is false. And a 4-bit base does not remove gradient checkpointing, because that saves activation memory, which is a separate thing, so option E is false.
**QLoRA** combines four ingredients, and three of the five options name them correctly. The correct set is A, B, and D.
Option A is right. The frozen base is stored in 4-bit NF4, whose buckets follow a Normal distribution because trained weights are roughly Gaussian. This beats uniform INT4 at the same bit width. Option B is right too. Double quantisation re-quantises the per-block scaling constants, saving about 0.37 bits per parameter, which matters at large scale.
Option D is right. Paged optimisers use NVIDIA unified memory to move optimiser moments to CPU when an out-of-memory spike looms, then page them back. Option C is the classic trap. The LoRA adapters stay in bf16, not 4-bit, so gradients flow cleanly into the low-rank factors.
Option E is also false. Gradient checkpointing saves activation memory, which is independent of weight bit-width. QLoRA runs still use it, exactly like full fine-tuning does.
Quick reference
TL;DR
QLoRA stacks four tricks: a 4-bit NF4 base, double quantisation of the constants, paged optimisers, and LoRA adapters kept in bf16.
Imagine shrinking a huge reference encyclopedia so it fits in your backpack. NF4 is a clever compression that keeps the most-used pages crispest, because most pages cluster around the middle. Double quantisation then compresses the little index cards that say how each section was shrunk. The encyclopedia itself you never write in; you keep frozen, compressed pages. Instead you carry a thin notebook of full-quality sticky notes (the adapters) where all your new learning happens. And if your desk runs out of room while studying, paged optimisers quietly move some scratch paper to a drawer and fetch it back later. Together these four tricks let you fine-tune a model that would normally need a server rack on a single gaming GPU.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You have ample GPU memory and want maximum quality from the fine-tune 4-bit quantisation introduces a small accuracy gap; plain LoRA or full fine-tuning in bf16 avoids it when memory is not the bottleneck.
Latency-critical serving where dequantisation overhead matters On-the-fly dequantisation of the 4-bit base adds compute per forward pass; a merged bf16 model serves faster.
The base model is small enough to fit comfortably in bf16 QLoRA's machinery only pays off when weight memory is the binding constraint, not for a 1B to 3B base.
You need to merge adapters and export a clean full-precision checkpoint Merging into a 4-bit base is lossy; you must dequantise first, which negates some of the storage savings.
Quick scan · 2–4 min
Everything important, quickly.
4 min: the four QLoRA components + why NF4 beats INT4 + what double quant compresses + why adapters stay bf16 + why gradient checkpointing is orthogonal.
Side-by-side
QLoRA component
What it touches
Precision / effect
NF4 base
Frozen base weights
4-bit, Normal-distributed buckets
Double quantisation
Per-block scaling constants
Saves about 0.37 bits per parameter
Paged optimisers
Adam optimiser state
Spills to CPU on memory spikes
LoRA adapters
Learnable low-rank factors
Stay in bf16 for stable gradients
Gradient checkpointing
Activations (orthogonal)
Still used; saves activation memory
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face PEFT plus bitsandbytes ships QLoRA as the default 4-bit recipe used to fine-tune Llama 3.1 70B on a single 48GB GPU.
2Unsloth and Axolotl both expose NF4 plus double quantisation as one-flag presets for QLoRA runs on consumer and rented GPUs.
3Open-weight tunes of DeepSeek V4 and Qwen3 distributed on the Hugging Face Hub are routinely produced with QLoRA on rented H100 instances.
4Together.ai and Modal Labs reference stacks rely on paged optimisers to keep long QLoRA jobs alive through transient memory spikes.
5Hugging Face TRL's SFTTrainer pairs QLoRA with gradient checkpointing as the standard memory-budget recipe for 2026 instruction tuning.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does NF4 use Normal-distributed quantile buckets instead of uniform spacing?
Sign in or go Pro to see the approach for this follow-up.
QWhat numerical problem would arise if you tried to keep the LoRA adapters in 4-bit too?
Sign in or go Pro to see the approach for this follow-up.
QHow does double quantisation arrive at roughly 0.37 bits per parameter of savings?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Thinking the LoRA adapters are also 4-bit, or that a 4-bit base removes the need for gradient checkpointing. Both confuse weight memory with where learning and activations actually live.