Stacking two unrelated LoRA adapters: most likely outcome

Attempt it

Quick reference

TL;DR

Merging two independently-trained LoRA adapters often hurts both tasks. Their deltas collide in shared weight directions. Route between adapters or train one joint adapter instead.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine two editors revising the same manuscript, working in separate rooms, neither seeing the other's marks. One rewrites for a legal audience; the other rewrites for children. Each edit makes sense alone. Now you stack both sets of changes onto one page. Sentences get cut twice, tone whipsaws, and the result reads worse than either editor's version on its own. That is what happens when you add two LoRA adapters trained apart: each assumed it owned the weights, so their changes fight over the same words. The fix is not to merge the pages. You keep both edited versions and pick the right one per request, or you give both editors the full brief up front so they revise together.

Key concepts

LoRA adapters LoRA composition Task arithmetic Multi-LoRA serving

Quick scan · 2–4 min

Everything important, quickly.

5 min: pick C and justify it, explain the additive-delta interference mechanism, debunk all three distractors, then walk routing versus joint training versus interference-aware merging.

Side-by-side

Strategy	What happens to quality	When to use
Naive merge (sum deltas)	Often regresses BOTH tasks for unrelated adapters	Only when adapters are near-identical or same warm start
Interference-aware merge (TIES, DARE)	Reduces but does not eliminate conflict	You need one fused checkpoint and can tolerate some loss
Joint training on union	Best quality, tasks share gradients	You control both datasets and can retrain
Route per request (multi-LoRA)	Each task stays at full adapter quality	Multi-tenant serving, many adapters, one base

Real-world examples

Real products, models, and research that use this idea.

vLLM and S-LoRA serve thousands of distinct LoRA adapters on one base model by routing per request, deliberately avoiding the merge and regress trap.
Predibase and Together.ai multi-tenant fine-tuning platforms keep customer adapters separate at inference rather than merging them into a shared base.
TIES-Merging and DARE are the standard interference-aware recipes when teams must fuse multiple task vectors into one Llama 4 or Qwen3 checkpoint.
Hugging Face PEFT exposes add_weighted_adapter with ties and dare_ties combination types precisely because naive summation degrades both source tasks.
Model-souping write-ups around open-weight merges in the Mistral and Llama communities show that only closely-related fine-tunes merge cleanly.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does low rank fail to prevent interference between two merged adapters?

Sign in or go Pro to see the approach for this follow-up.

QHow do TIES-Merging and DARE reduce the destructive interference of a naive sum?

Sign in or go Pro to see the approach for this follow-up.

QWhen is merging two adapters actually safe, and why?

Sign in or go Pro to see the approach for this follow-up.

QHow would multi-LoRA serving like S-LoRA avoid the merge problem entirely?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.