Can you stack two LoRA adapters? What's the failure mode?
Suppose you have two LoRA adapters trained independently: one on a code corpus, one on a medical-records corpus. Can you 'stack' them by summing their BA matrices into a single combined adapter? What works, what breaks, and what's the safer composition pattern?
Summing two LoRA adapters is valid linear algebra. Related tasks compose; orthogonal ones interfere. Prefer a router or multi-LoRA serving over a physical merge.
Each LoRA adapter is a small nudge you add on top of a frozen model, and nudges add up like vectors. If two nudges point roughly the same way, like Python skill and JavaScript skill, adding them gives a bigger push in a useful direction. If they point in totally different directions, like coding skill and medical skill, the sum points somewhere in between that is good at neither. So the math always lets you add them, but the result is only good when the two skills are related. The safer trick is to keep both adapters separate and let a tiny chooser pick the right one for each question, instead of permanently blending them into one.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5 min: linear-sum validity + rank bound + related vs orthogonal composition + why rank worsens interference + router, joint training, and multi-LoRA serving as safer patterns.
| Composition pattern | When it works | Main risk |
|---|---|---|
| Naive sum of BA matrices | Related tasks with aligned updates | Destructive interference on orthogonal tasks |
| Weighted task arithmetic | Close tasks needing balance control | Coefficients are dataset specific and brittle |
| Mixture of LoRA router | Distinct tasks, one fires per request | Router misrouting; extra training and latency |
| Joint training on union | Both skills must co-fire in one answer | Cost of a fresh training run on combined data |
| Multi-LoRA serving | Multi-tenant, per-request adapter choice | No single merged artifact; serving complexity |
Real products, models, and research that use this idea.
- vLLM multi-LoRA and S-LoRA serve thousands of distinct adapters on one base model, applying the right adapter per request without ever merging weights.
- Hugging Face PEFT exposes add_weighted_adapter, letting you sum adapters with explicit coefficients to test composition before committing to a merge.
- LoRAHub composes task adapters via learned weighting for few-shot transfer, an explicit weighted-arithmetic approach rather than a naive sum.
- Predibase and Together.ai run multi-tenant LoRA fleets on Llama 4 bases where each customer's adapter stays isolated at inference, sidestepping interference.
- Apple's on-device adapter approach swaps task-specific adapters over a shared foundation model rather than merging them into one set of weights.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you quantify whether two adapters will interfere before merging them?
QWhy does destructive interference get worse as LoRA rank increases?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Assuming a summed adapter always preserves both skills. Destructive interference is the default for unrelated tasks, and it gets worse as rank rises.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.