After calling `merge_and_unload()` on a PEFT model, what does the returned object look like?

**LoRA** keeps the base model frozen and trains a small adapter, a pair of matrices A and B, whose product is added to the base weight at every forward pass. The actual computation at inference looks like base times input plus a scaled adapter product times input. **merge_and_unload** changes that shape. The method walks every LoRA-wrapped linear in the model, computes the new merged weight as the original base weight plus the scaled adapter product, and writes that single matrix back into the layer. After the merge it removes the LoRA wrapper, leaving a plain linear layer with one weight matrix. The returned object looks like a regular AutoModelForCausalLM. It has no PEFT machinery and no adapter state. Every forward pass is one matmul per linear, exactly like a fully fine-tuned model would be. The benefit is deployment simplicity. Inference servers that do not know about LoRA can load and serve the merged model with no special handling. The cost is flexibility: you can no longer hot-swap adapters or compose multiple adapters at runtime, because the adapter is gone.

**merge_and_unload** does exactly what its two verbs suggest: it merges the adapter delta into the base weight at every wrapped layer, then unloads the adapter wrapper. For each LoRA-wrapped linear, the merged weight is: $$W' = W_0 + \tfrac{\alpha}{r}\, B \, A$$ The scaled adapter product is computed once, added to the base weight in place, and the adapter modules A and B are removed from the module tree. After the call you hold a plain `AutoModelForCausalLM`-shaped object with no PEFT runtime, no adapter state, and no per-step low-rank add at forward time. The deployment win is concrete. Serving stacks like vLLM, TGI, and TensorRT-LLM can load the merged checkpoint directly with no awareness of LoRA. The forward pass is one matmul per linear, which is what their optimised kernels expect. Memory at inference matches the base model exactly, since there is no adapter overhead. The trade off is flexibility. While the wrapper is in place you can hot-swap adapters, compose multiple adapters with set_adapter, or disable them with disable_adapter for A/B testing. After merge_and_unload all of that is gone. The merged checkpoint is a single self-contained artefact, useful for serving and packaging, less useful when you need adapter agility.