Merging a LoRA back into an NF4-quantized base hits one subtle problem: name it and explain why

Short answer·Hard·4.0 · 0·~3 min·Relevant atDatabricks Meta OpenAI

Attempt it

QLoRA stores the base weights in NF4 (4-bit) and trains LoRA adapters in bf16 on top. After training, you want to merge the LoRA delta back into the base so inference can use a plain weight tensor with no adapter overhead. Name the subtle problem you hit, explain the underlying type-arithmetic issue, and give the practical merge pattern teams actually use.

Free · 2 AI evals / day

The subtle problem is that you cannot add a bf16 tensor directly to an NF4 tensor. **NF4** is not a numeric format; it is a 4-bit code into a small codebook of sixteen non-uniformly spaced values. The actual weight is recovered by looking up the codebook entry and multiplying by a per-block scale. Arithmetic on NF4 codes is not defined. So a merge cannot be one tensor addition. The pipeline has to dequantize the NF4 base back to bf16, add the bf16 LoRA delta, and then decide what to do with the result. Two choices, both lossy in different ways. Choice one: keep the merged tensor in bf16. Numerics are clean and the model behaves exactly as trained. The cost is that you have given up the 4x memory win that motivated QLoRA in the first place. The base now takes four times the disk and four times the VRAM at inference. Choice two: re-quantize the merged tensor back to NF4. Memory is recovered but the codebook was fit to the base weight distribution, not to base plus delta. The merged weights round to different codebook entries than the unmerged base did, and the model's behaviour can drift noticeably from what you trained. The practical pattern: load the base in bf16 for the merge step, add the delta in bf16, save the merged checkpoint as bf16, and create a separately quantized NF4 build as a deployment artifact. One extra checkpoint format, clean numerics, clean serving.

**You cannot add a bf16 delta to an NF4 codebook.** NF4 stores each weight as one of sixteen non-uniformly spaced codebook entries scaled by a per-block factor; the format encodes positions in a fitted codebook, not numeric values you can do arithmetic on. Any merge requires dequantizing the base back to a higher precision first. The canonical pipeline is dequant, add, then resolve. Dequantize the NF4 base block by block into bf16. Add the bf16 LoRA delta. Now choose between two lossy resolutions. Resolution one is to keep the merged tensor in bf16. The numerics match what you trained on; the model behaves as the adapter loaded version would. The cost is that you have given up QLoRA's 4x memory win, the very reason you used QLoRA. Disk and VRAM at inference both jump by 4x. Resolution two is to re-quantize the merged tensor back to NF4. Memory is recovered but the original NF4 codebook was fit to the base weight distribution, not to base plus delta. After the addition, the per-block weight distribution has shifted, and the closest codebook entries no longer cover the new values cleanly. The merged weights round to different codes than the unmerged base did, and quality can drift noticeably, especially on harder tasks where the LoRA delta carried fine-grained behaviour. The pattern most teams use: load the base in bf16 for the merge step, add the delta in bf16, save the merged checkpoint as bf16, and produce a separately quantized NF4 build as a deployment artifact. You accept one extra checkpoint format in exchange for clean numerics on the training path and a clean deployment build on the serving path. The two are decoupled, and the lossiness lives entirely in the deployment quantization step where it can be measured and bounded against eval.

Quick reference

TL;DR

You can't add a bf16 delta directly to an NF4 codebook; either dequant-add and keep bf16 (loses memory win) or dequant-add and requant (distorts the delta), so teams merge into a bf16 base and ship a separate NF4 build.

Memory aid

Easy to grasp

Picture a chef who organises a pantry with about a dozen labelled jars that approximate every spice they ever use. A guest hands them a new spice blend in a measuring cup. They cannot just pour the blend into one of the jars; the jars hold the codes, not the spices themselves. The chef has to pull a few jars off the shelf, mix them up to make the real spice, add the guest's blend, and then either keep the mixture in the measuring cup or pour it back into the closest jar, which will round the new blend to whatever pre-set ratio is nearest. Keeping the cup is accurate but takes more shelf space. Rounding into a jar saves space but loses the guest's exact recipe. Teams take the cup home for accuracy and brew a fresh small-jar version separately for serving.

Key concepts

QLoRA LoRA NF4 quantization Adapter merging Double quantization

Detailed answer & concept explanation

~9 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

7 min: explain why NF4 codes cannot accept a bf16 add directly, walk through dequant, add, resolve with both lossy choices, prescribe the bf16-merge plus separate NF4-deploy pattern, and call out merge_and_unload behaviour plus double quantization pitfalls.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face PEFT documents merge_and_unload behaviour on 4-bit-loaded models, with explicit warnings that the output precision is not 4-bit and that re-quantization must be run separately.
QLoRA paper (Dettmers et al. 2023) specifies the dequantize then add pattern and shows that fine-tuned NF4 base plus bf16 adapter behaves equivalently to bf16 base plus bf16 adapter at inference.
bitsandbytes' quantize_4bit and dequantize_4bit utilities expose the codebook and per-block scales explicitly, making custom merge pipelines feasible for production teams.
Modern serving stacks like vLLM and TGI accept bf16 merged checkpoints and run their own AWQ or GPTQ quantization pass as a separate deployment step, decoupling the merge artifact from the serving artifact.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you measure whether re-quantization after merge actually hurts quality on your task?

QWhy does double quantization complicate the merge pipeline, and how do you handle it?

QWhen does adapter hot-swap inference beat merging entirely?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Calling merge_and_unload on a model loaded in 4-bit and expecting a clean NF4 output. The merge dequantizes silently or fails; either way the deployment artifact is not what you think it is.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.