Merging a LoRA back into an NF4 quantized base hits one subtle problem: name it and explain why

Short answer·Hard·4.0 · 0·~3 min·Asked atC3 Ai Persistent Redis·Relevant atAnthropic Databricks Meta OpenAI

Attempt it

QLoRA stores the base weights in NF4 (4-bit) and trains LoRA adapters in bf16 on top. After training, you want to merge the LoRA delta back into the base so inference can use a plain weight tensor with no adapter overhead. Name the subtle problem you hit, explain the underlying type arithmetic issue, and give the practical merge pattern teams actually use.

Free · 2 AI evals / day

The subtle problem is that you cannot add a bf16 tensor directly to an NF4 tensor. **NF4** is not a numeric format; it is a 4-bit code into a small codebook of sixteen non-uniformly spaced values. The actual weight is recovered by looking up the codebook entry and multiplying by a per-block scale. Arithmetic on NF4 codes is not defined. So a merge cannot be one tensor addition. The pipeline has to dequantize the NF4 base back to bf16, add the bf16 LoRA delta, and then decide what to do with the result. Two choices, both lossy in different ways. Choice one: keep the merged tensor in bf16. Numerics are clean and the model behaves exactly as trained. The cost is that you have given up the 4x memory win that motivated QLoRA in the first place. The base now takes four times the disk and four times the VRAM at inference. Choice two: re-quantize the merged tensor back to NF4. Memory is recovered but the codebook was fit to the base weight distribution, not to base plus delta. The merged weights round to different codebook entries than the unmerged base did, and the model's behaviour can drift noticeably from what you trained. The practical pattern: load the base in bf16 for the merge step, add the delta in bf16, save the merged checkpoint as bf16, and create a separately quantized NF4 build as a deployment artifact. One extra checkpoint format, clean numerics, clean serving.

**You cannot add a bf16 delta to an NF4 codebook.** NF4 stores each weight as one of sixteen non-uniformly spaced codebook entries scaled by a per-block factor; the format encodes positions in a fitted codebook, not numeric values you can do arithmetic on. Any merge requires dequantizing the base back to a higher precision first. The canonical pipeline is dequant, add, then resolve. Dequantize the NF4 base block by block into bf16. Add the bf16 LoRA delta. Now choose between two lossy resolutions. Resolution one is to keep the merged tensor in bf16. The numerics match what you trained on; the model behaves as the adapter loaded version would. The cost is that you have given up QLoRA's 4x memory win, the very reason you used QLoRA. Disk and VRAM at inference both jump by 4x. Resolution two is to re-quantize the merged tensor back to NF4. Memory is recovered but the original NF4 codebook was fit to the base weight distribution, not to base plus delta. After the addition, the per-block weight distribution has shifted, and the closest codebook entries no longer cover the new values cleanly. The merged weights round to different codes than the unmerged base did, and quality can drift noticeably, especially on harder tasks where the LoRA delta carried fine-grained behaviour. The pattern most teams use: load the base in bf16 for the merge step, add the delta in bf16, save the merged checkpoint as bf16, and produce a separately quantized NF4 build as a deployment artifact. You accept one extra checkpoint format in exchange for clean numerics on the training path and a clean deployment build on the serving path. The two are decoupled, and the lossiness lives entirely in the deployment quantization step where it can be measured and bounded against eval.

TL;DR

You can't add a bf16 delta directly to an NF4 codebook; either dequant-add and keep bf16 (loses memory win) or dequant-add and requant (distorts the delta), so teams merge into a bf16 base and ship a separate NF4 build.

Memory aid

Easy to grasp

Picture a chef who organises a pantry with about a dozen labelled jars that approximate every spice they ever use. A guest hands them a new spice blend in a measuring cup. They cannot just pour the blend into one of the jars; the jars hold the codes, not the spices themselves. The chef has to pull a few jars off the shelf, mix them up to make the real spice, add the guest's blend, and then either keep the mixture in the measuring cup or pour it back into the closest jar, which will round the new blend to whatever pre-set ratio is nearest. Keeping the cup is accurate but takes more shelf space. Rounding into a jar saves space but loses the guest's exact recipe. Teams take the cup home for accuracy and brew a fresh small-jar version separately for serving.

Key concepts

QLoRA LoRA NF4 quantization Adapter merging Double quantization

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Adapter merging looks like a one-line operation in the API. You call merge_and_unload on a PEFT model, the LoRA matrices fold into the base, and you ship the result. The reality, when the base is held in NF4, is that the one-line call hides a precision dance with two lossy resolution choices and a serving artifact that almost nobody verifies as carefully as they should.

The core obstacle is that NF4 is not a numeric format. It is a 4-bit index into a sixteen-entry codebook of non-uniformly spaced reconstruction values, multiplied per block by a separately stored scale. The codes are positions, not numbers. There is no defined arithmetic over NF4 codes, no addition operator, no rounding rule that maps an arbitrary bf16 value to a code in a single step. Any merge that touches an NF4 weight must first reconstruct it back to bf16, do the math in bf16, and then decide what precision to persist.

This question is testing whether the candidate has actually thought through that decision rather than trusting the API to do something sensible. The decision matters because it forces a tradeoff between two costs that QLoRA users typically want to avoid simultaneously: the memory cost that motivated QLoRA in the first place, and the quality cost of compounding rounding error through a second quantization pass. The answer is not 'pick one'; the answer is 'separate the training artifact from the serving artifact and let each carry its own precision profile'.

Why NF4 cannot accept a bf16 addition

NF4 was introduced in the QLoRA paper as a 4-bit format tuned for the empirical distribution of pretrained transformer weights. The format works as follows. Weights within a small block, typically 64 elements, are divided by a per-block fp16 or fp32 scale so they land in a normalised range around zero. Each normalised value is then snapped to the nearest entry in a fixed codebook of sixteen reconstruction levels, with the levels placed at the quantiles of a standard normal distribution rather than at uniformly spaced points. Each weight is stored as the 4-bit codebook index. To reconstruct a weight, look up the codebook entry by index and multiply by the block scale.

The format is excellent for memory efficiency because the codebook is fit to the actual distribution of weights, not to an arbitrary uniform grid. A weight near zero has many codebook entries around it; a weight in the tails has fewer, but the tails are sparse so the average rounding error stays small.

The format is also closed to arithmetic. Adding two NF4 codes is meaningless because the codes are positions in a lookup table, not numbers. Adding a bf16 value to an NF4 code requires first turning the code back into a bf16 value, which is a lookup and multiply operation. There is no shortcut. Any merge involving an NF4 tensor must include that dequantization step explicitly, and any code that pretends otherwise is doing the dequant silently underneath.

This is why naive expectations of 'merge in place on NF4' do not hold. The base tensor is held in 4-bit; the LoRA delta is bf16; their sum lives in bf16 by necessity. The question then becomes what to do with that bf16 sum, and that is where the two lossy resolutions enter.

Resolution one: keep the merged tensor in bf16

Resolution two: re-quantize the merged tensor back to NF4

The pattern teams actually use

Implementation pitfalls worth knowing

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Hugging Face PEFT documents merge_and_unload behaviour on 4-bit-loaded models, with explicit warnings that the output precision is not 4-bit and that re-quantization must be run separately.
QLoRA paper (Dettmers et al. 2023) specifies the dequantize then add pattern and shows that fine-tuned NF4 base plus bf16 adapter behaves equivalently to bf16 base plus bf16 adapter at inference.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you measure whether re-quantization after merge actually hurts quality on your task?

Run the same eval on three artifacts: the adapter-loaded model in bf16+NF4, the merged bf16 model, and the re-quantized NF4 merged model. The first to second gap measures merge lossiness; the second to third gap measures re-quantization lossiness; act on whichever is material.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Calling merge_and_unload on a model loaded in 4-bit and expecting a clean NF4 output. The merge dequantizes silently or fails; either way the deployment artifact is not what you think it is.

60 second bullets to scan on the way to the call.

Why NF4 codes cannot be added directly to bf16 tensors
How dequantize, add, resolve structures the merge pipeline

Primary sources. Browse if you want the original framing.