Merge LoRA into the base vs keep it as a swappable adapter: when do you pick which?
After training a LoRA adapter, you can either merge it into the base model (so the deployed checkpoint is a single fused weight tensor) or keep it as a swappable adapter loaded alongside the base. What's the serving implication of each choice, and when do you pick which?
Merge a finalised adapter for single-tenant serving with zero overhead. Keep it swappable when one base must serve many fine-tunes or you A/B often.
A LoRA adapter is like a small accent coach bolted onto a singer. Merging means you retrain the singer once so the new accent is permanent, after that there is nothing extra to carry, they just sing. Keeping it swappable means the coach stands beside the singer and whispers tweaks live, so you can swap in a different coach between songs without hiring a new singer. One singer plus many coaches is cheap when you need lots of styles. But the whispering adds a tiny delay every line. If you only ever need one accent forever, bake it in. If you run a karaoke bar serving hundreds of styles, keep the coaches separate and swap them on demand.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: the merged fused-weight formula + why merge removes overhead + how swappable applies a residual add + S-LoRA and vLLM multi-tenant batching + the single versus multi-tenant decision + the post-merge quantisation trap.
| Concern | Merged into base | Swappable adapter |
|---|---|---|
| Deployed artifact | Single fused weight tensor per layer | Frozen base plus small adapter file |
| Per-step overhead | None, identical to vanilla checkpoint | Residual add plus per-request routing |
| Multi-fine-tune hosting | One full checkpoint per task | Thousands of adapters on one base |
| Swap or revert | Reload the whole model | Change one small file |
| Best for | Single-tenant, finalised, latency-critical | Multi-tenant, frequent A/B, tenant isolation |
Real products, models, and research that use this idea.
- Predibase and Together.ai serve thousands of customer LoRA adapters on shared base models using S-LoRA style multi-LoRA batching, so each tenant pays adapter storage not full-model hosting.
- vLLM ships built-in multi-LoRA support that routes per-request adapters against one Llama 4 base, used by teams running many specialised fine-tunes behind a single endpoint.
- Hugging Face PEFT exposes merge_and_unload() to fuse a finalised adapter into the base for clean single-purpose deployment with no runtime adapter path.
- OpenAI and Anthropic hosted fine-tuning on GPT-5.5 and Claude Opus 4.7 abstract the merge versus swap choice away, but internally rely on multi-tenant adapter serving to keep per fine-tune cost low.
- Replicate hosts community fine-tunes by keeping adapters swappable on shared GPUs, merging only when a model graduates to a dedicated always-on deployment.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does S-LoRA serve thousands of adapters on one base without a separate forward pass per adapter?
QWhy might a merged then quantised checkpoint differ from the unmerged adapter applied at full precision?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Merging a still-iterating adapter and losing the ability to revert or swap, or paying multi-LoRA routing overhead when you only ever serve one finalised fine-tune.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.