Can you stack two LoRA adapters? What's the failure mode?
Suppose you have two LoRA adapters trained independently: one on a code corpus, one on a medical records corpus. Can you 'stack' them by summing their BA matrices into a single combined adapter? What works, what breaks, and what's the safer composition pattern?
Summing two LoRA adapters is valid linear algebra. Related tasks compose; orthogonal ones interfere. Prefer a router or multi-LoRA serving over a physical merge.
Each LoRA adapter is a small nudge you add on top of a frozen model, and nudges add up like vectors. If two nudges point roughly the same way, like Python skill and JavaScript skill, adding them gives a bigger push in a useful direction. If they point in totally different directions, like coding skill and medical skill, the sum points somewhere in between that is good at neither. So the math always lets you add them, but the result is only good when the two skills are related. The safer trick is to keep both adapters separate and let a tiny chooser pick the right one for each question, instead of permanently blending them into one.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This question looks like a yes or no algebra check, and the algebra part is genuinely easy. A LoRA update is linear in its B and A matrices, so summing two adapters is always a well-defined operation that yields another low-rank update. Any candidate can clear that bar. The reason it is a staff-level question is that the interesting answer is about geometry, interference, and the production patterns that avoid the whole problem.
The trap is to stop at 'yes, you can add them' and assume the sum preserves both skills. It does not, in general. Two adapters trained independently have no knowledge of each other. When their tasks are related they tend to reinforce; when their tasks are orthogonal they tend to collide. The same operation produces a clean merge in one case and a degraded model in the other, and the difference is entirely about how the two updates sit in weight space.
It helps to separate three distinct claims that the naive answer tends to blur together. The first is that the merge is computable, which is always true. The second is that the merge preserves each individual skill, which holds only for aligned tasks. The third is that the merge lets both skills co-fire usefully in one response, which is the hardest property and almost never falls out of a sum. Interviewers probe this question precisely because candidates collapse all three into the first.
This deep dive walks the full arc: the math of stacking, why related tasks compose while orthogonal ones interfere, why rank amplifies the effect, and the three production patterns that sidestep destructive merges. The framing to carry through is that merging is a last resort you reach for only after measuring clean composition, not a default you assume works.
The math: stacking is always valid
A LoRA adapter replaces a full weight update with a low-rank factorisation. For a frozen base weight, the adapted weight is the base plus a scaled product of two small matrices. Because that correction is purely additive, two adapters on the same base simply add:
This is always well-defined. The combined correction is still low-rank, bounded above by the sum of the two ranks. So at the level of linear algebra there is no obstacle whatsoever to merging adapters, and tools like the PEFT add_weighted_adapter call expose exactly this operation with per-adapter coefficients.
One subtlety to flag: even the rank bound is an upper bound, not an equality. If the two B·A products happen to share column space, the effective rank of the sum is lower than the two ranks added together. That is exactly the aligned-task case, and it hints at why related adapters merge gracefully while orthogonal ones do not.
The catch is that being well-defined is not the same as being good. The sum is a valid weight matrix, but whether it behaves like a model that has both skills is an empirical property of the two tasks, not a guarantee from the algebra. That gap between mathematical validity and behavioral quality is the whole question.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Composition pattern | When it works | Main risk |
|---|---|---|
| Naive sum of BA matrices | Related tasks with aligned updates | Destructive interference on orthogonal tasks |
| Weighted task arithmetic | Close tasks needing balance control | Coefficients are dataset specific and brittle |
| Mixture of LoRA router | Distinct tasks, one fires per request | Router misrouting; extra training and latency |
| Joint training on union | Both skills must co-fire in one answer | Cost of a fresh training run on combined data |
| Multi-LoRA serving | Multi-tenant, per-request adapter choice | No single merged artifact; serving complexity |
Real products, models, and research that use this idea.
- vLLM multi-LoRA and S-LoRA serve thousands of distinct adapters on one base model, applying the right adapter per request without ever merging weights.
- Hugging Face PEFT exposes add_weighted_adapter, letting you sum adapters with explicit coefficients to test composition before committing to a merge.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you quantify whether two adapters will interfere before merging them?
Look at the overlap of their update subspaces. Compare the column spaces of the two B·A products via subspace angles or cosine similarity of flattened deltas, and eval composition on a held-out set for both tasks.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming a summed adapter always preserves both skills. Destructive interference is the default for unrelated tasks, and it gets worse as rank rises.
60 second bullets to scan on the way to the call.
Why summing BA matrices is always mathematically valid
The rank bound on the combined adapter
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.