After calling `merge_and_unload()` on a PEFT model, what does the returned object look like?
merge_and_unload folds the LoRA delta into each base weight and strips the adapter wrapper, returning a plain transformer model with a single matmul per linear and no PEFT runtime.
Picture a violin with a small electronic effects pedal attached by a cable. While the cable is plugged in, the sound is the violin plus the pedal effect every time you play. The pedal is the adapter. merge_and_unload is like baking the pedal effect into the violin's wood itself, then unplugging and discarding the pedal. The instrument now plays the combined sound on its own, no pedal, no cable. The downside is that you can no longer swap pedals to get a different effect; the violin is permanently the merged version. The upside is that anyone can pick it up and play without needing to know about pedals or cables. That is exactly what merge_and_unload does to a PEFT model: it fuses the adapter into the weights and removes the wrapper.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
PEFT methods like LoRA produce a two-piece artefact: the frozen base weights and a small adapter that modifies their behavior at forward time. For training and rapid experimentation this two-piece structure is exactly what you want, because you can swap adapters, compose them, or disable them without touching the base. For production serving the same structure is often the wrong choice, because most serving stacks expect a single set of weights and a forward pass that is one matmul per linear.
merge_and_unload is the bridge between those two worlds. It collapses the two-piece artefact into a single set of weights with the adapter's effect permanently fused in, and removes the PEFT wrapper that made the two-piece structure work at training time. The returned object looks and behaves like a plain transformer model that happened to be fully fine-tuned, even though the underlying weights came from a frozen base plus a low-rank adapter.
This deep dive walks through the precise operation, the math of the merged weight, what happens to the PEFT runtime, the deployment scenarios that motivate the merge, the flexibility you give up, and the quantised base wrinkle that catches teams off guard.
What the method does, layer by layer
When you call merge_and_unload on a PeftModel, the method walks the module tree and operates on every LoRA-wrapped linear (or convolution, or whatever module class the adapter targets). For each wrapped module it does two things.
The merge step
The wrapped linear has access to three tensors: the original base weight W_0, and the two adapter matrices A and B. The forward pass at training time computes W_0 x + (alpha / r) * B A x. The merge step computes the new weight directly:
This is one matrix multiplication between B and A, scaled by alpha over r, added to the base weight. The result is a single matrix with the same shape as the original base weight. The operation is in-place: the wrapped layer's weight tensor is overwritten with W'.
There is no approximation here. The merged weight exactly reproduces the forward pass of the wrapped layer for any input x, because (W_0 + (alpha / r) * B A) x = W_0 x + (alpha / r) * B A x. Quality at inference is identical to the unmerged model.
The unload step
After the merge, the adapter matrices A and B are no longer needed. The unload step removes them from the module tree and replaces the LoRA-wrapped linear with a plain linear that holds the merged weight W'. The returned object is no longer a PeftModel; it is whatever the base class was before PEFT wrapped it, typically AutoModelForCausalLM.
The state dict of the returned object contains a single weight per linear, no adapter sidecar. save_pretrained writes a single set of files that any standard transformer loader can read.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Hugging Face PEFT exposes merge_and_unload on PeftModel and ships it as the canonical step before exporting a Llama 4 or Mistral checkpoint for serving with vLLM.
- Axolotl provides a merge script that wraps merge_and_unload and is the standard final step in the QLoRA recipe for Gemma 4 and DeepSeek V4 distillations.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does merge_and_unload interact with 4-bit quantised bases like those used in QLoRA?
Discuss the need to dequantise the base before computing the merged weight in higher precision, then optionally re-quantise; explain why teams sometimes keep the wrapper rather than merge into a quantised base.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking merge_and_unload keeps the adapter for fast hot-swap later. It does the opposite: it fuses the delta into the base and removes the adapter, trading flexibility for a plain serving stack.
60 second bullets to scan on the way to the call.
What merge_and_unload does to each wrapped linear
The formula for the merged weight
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.