After calling merge_and_unload() on a PEFT model, what does the returned object look like?
merge_and_unload folds the LoRA delta into each base weight and strips the adapter wrapper, returning a plain transformer model with a single matmul per linear and no PEFT runtime.
Picture a violin with a small electronic effects pedal attached by a cable. While the cable is plugged in, the sound is the violin plus the pedal effect every time you play. The pedal is the adapter. merge_and_unload is like baking the pedal effect into the violin's wood itself, then unplugging and discarding the pedal. The instrument now plays the combined sound on its own, no pedal, no cable. The downside is that you can no longer swap pedals to get a different effect; the violin is permanently the merged version. The upside is that anyone can pick it up and play without needing to know about pedals or cables. That is exactly what merge_and_unload does to a PEFT model: it fuses the adapter into the weights and removes the wrapper.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
PEFT methods like LoRA produce a two-piece artefact: the frozen base weights and a small adapter that modifies their behavior at forward time. For training and rapid experimentation this two-piece structure is what you want. You can swap adapters, compose them, or disable them without touching the base. For production serving the same structure is often the wrong choice. Most serving stacks expect a single set of weights and a forward pass that is one matmul per linear.
merge_and_unload is the bridge between those two worlds. It collapses the two-piece artefact into a single set of weights with the adapter's effect permanently fused in. It also removes the PEFT wrapper that made the two-piece structure work at training time. The returned object looks and behaves like a plain transformer model that happened to be fully fine-tuned. The underlying weights still came from a frozen base plus a low-rank adapter.
What the method does, layer by layer
When you call merge_and_unload on a PeftModel, the method walks the module tree. It operates on every LoRA-wrapped linear (or convolution, or whatever module class the adapter targets). For each wrapped module it does two things.
The merge step
The wrapped linear has access to three tensors: the original base weight W_0, and the two adapter matrices A and B. The forward pass at training time computes W_0 x + (alpha / r) * B A x. The merge step computes the new weight directly:
This is one matrix multiplication between B and A, scaled by alpha over r, added to the base weight. The result is a single matrix with the same shape as the original base weight. The operation is in place: the wrapped layer's weight tensor is overwritten with W'.
There is no approximation here, apart from floating-point rounding (and re-quantisation on a 4-bit base). The merged weight reproduces the forward pass of the wrapped layer for any input x, because (W_0 + (alpha / r) * B A) x = W_0 x + (alpha / r) * B A x. Quality at inference is identical to the unmerged model.
The unload step
After the merge, the adapter matrices A and B are no longer needed. The unload step removes them from the module tree and replaces the LoRA-wrapped linear with a plain linear that holds the merged weight W'. The returned object is no longer a PeftModel. It is whatever the base class was before PEFT wrapped it, typically AutoModelForCausalLM.
The state dict of the returned object contains a single weight per linear, no adapter sidecar. save_pretrained writes a single set of files that any standard transformer loader can read.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Hugging Face PEFT exposes merge_and_unload on PeftModel to turn a base plus adapter into a standalone model, for example before serving with a stack that has no LoRA support.
- Axolotl provides a merge-lora command that merges a trained LoRA or QLoRA adapter into its base model and saves the result.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does merge_and_unload interact with 4-bit quantised bases like those used in QLoRA?
Discuss the need to dequantise the base before computing the merged weight in higher precision, then optionally re-quantise. Explain why teams sometimes keep the wrapper rather than merge into a quantised base.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking merge_and_unload keeps the adapter for fast hot-swap later. It does the opposite: it fuses the delta into the base and removes the adapter, trading flexibility for a plain serving stack.
60 second bullets to scan on the way to the call.
What merge_and_unload does to each wrapped linear
The formula for the merged weight
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.