Merge LoRA into the base vs keep it as a swappable adapter: when do you pick which?

Attempt it

After training a LoRA adapter, you can either merge it into the base model (so the deployed checkpoint is a single fused weight tensor) or keep it as a swappable adapter loaded alongside the base. What's the serving implication of each choice, and when do you pick which?

Free · 2 AI evals / day

Quick reference

TL;DR

Merge a finalised adapter for single-tenant serving with zero overhead. Keep it swappable when one base must serve many fine-tunes or you A/B often.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

A LoRA adapter is like a small accent coach bolted onto a singer. Merging means you retrain the singer once so the new accent is permanent, after that there is nothing extra to carry, they just sing. Keeping it swappable means the coach stands beside the singer and whispers tweaks live, so you can swap in a different coach between songs without hiring a new singer. One singer plus many coaches is cheap when you need lots of styles. But the whispering adds a tiny delay every line. If you only ever need one accent forever, bake it in. If you run a karaoke bar serving hundreds of styles, keep the coaches separate and swap them on demand.

Key concepts

LoRA adapters LLM serving Adapter merge Multi-LoRA serving

Quick scan · 2–4 min

Everything important, quickly.

4 min: the merged fused-weight formula + why merge removes overhead + how swappable applies a residual add + S-LoRA and vLLM multi-tenant batching + the single versus multi-tenant decision + the post-merge quantisation trap.

Side-by-side

Concern	Merged into base	Swappable adapter
Deployed artifact	Single fused weight tensor per layer	Frozen base plus small adapter file
Per-step overhead	None, identical to vanilla checkpoint	Residual add plus per-request routing
Multi-fine-tune hosting	One full checkpoint per task	Thousands of adapters on one base
Swap or revert	Reload the whole model	Change one small file
Best for	Single-tenant, finalised, latency-critical	Multi-tenant, frequent A/B, tenant isolation

Real-world examples

Real products, models, and research that use this idea.

Predibase and Together.ai serve thousands of customer LoRA adapters on shared base models using S-LoRA style multi-LoRA batching, so each tenant pays adapter storage not full-model hosting.
vLLM ships built-in multi-LoRA support that routes per-request adapters against one Llama 4 base, used by teams running many specialised fine-tunes behind a single endpoint.
Hugging Face PEFT exposes merge_and_unload() to fuse a finalised adapter into the base for clean single-purpose deployment with no runtime adapter path.
OpenAI and Anthropic hosted fine-tuning on GPT-5.5 and Claude Opus 4.7 abstract the merge versus swap choice away, but internally rely on multi-tenant adapter serving to keep per fine-tune cost low.
Replicate hosts community fine-tunes by keeping adapters swappable on shared GPUs, merging only when a model graduates to a dedicated always-on deployment.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does S-LoRA serve thousands of adapters on one base without a separate forward pass per adapter?

Sign in or go Pro to see the approach for this follow-up.

QWhy might a merged then quantised checkpoint differ from the unmerged adapter applied at full precision?

Sign in or go Pro to see the approach for this follow-up.

QHow would you A/B test two adapters in production with minimal serving disruption?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.