What does bnb_4bit_compute_dtype actually control in a QLoRA config?

In **QLoRA** the base model is loaded with weights packed at 4 bits each. That is great for memory but useless for math, because GPUs do not have 4-bit matmul instructions. So the library does a two-step dance at every layer. It keeps the weights on the GPU in 4-bit form, and right before each multiplication it expands a small block of weights back to a normal precision like bf16. The matmul runs in bf16, and the result is added to whatever comes next. The flag `bnb_4bit_compute_dtype` picks which precision that expansion targets. People almost always set it to bf16 on H100 and A100 GPUs because bf16 is fast, has wide dynamic range, and is what the rest of the training stack already uses. The weights themselves never change format on disk or in memory between matmuls. They stay 4-bit. Only the temporary expanded copy lives at the compute dtype. So you keep all the memory savings of 4-bit storage and almost all the numerical quality of bf16 compute, which is the whole point of QLoRA.

**`bnb_4bit_compute_dtype`** controls the precision used inside the matmul after on the fly dequantization of the 4-bit base weights. Storage stays 4-bit; compute happens at the chosen dtype. QLoRA loads the frozen base in NF4 to slash GPU memory roughly four times versus bf16. But GPUs do not execute 4-bit matmuls natively, so bitsandbytes dequantizes each block of weights to the compute dtype just before the multiplication, runs the matmul there, and discards the expanded copy. The on-disk and in-memory weight layout never changes. The practical pick is bf16 on Ampere and Hopper, fp16 on older Volta and Turing, and fp32 if you are debugging numerical issues. Bf16 is the default in almost every QLoRA recipe because it matches the rest of the mixed-precision stack and has the dynamic range needed for transformer gradients. Watch out for one trap. The flag does not change activations, LoRA adapter dtype, or optimizer state. Those are separate settings. Picking bf16 here only affects what dtype the dequantized base weights live in for the duration of one matmul.