How does S-LoRA serve thousands of LoRA adapters concurrently on a single base model?

Attempt it

Explain how S-LoRA serves thousands of distinct LoRA adapters from one shared base model, including how the forward pass handles a batch where each request targets a different adapter.

Free · 2 AI evals / day

Quick reference

TL;DR

Keep one base model resident, store thousands of small LoRA adapters in a paged pool, and apply each request's adapter inside one batched call via a segmented gather-scatter matmul.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a print shop with one giant printing press and thousands of small custom stamps. The press is expensive, so you keep exactly one and never swap it. Each customer brings a tiny stamp that tweaks the output just for them. You keep the popular stamps on the desk and the rest in a drawer, fetching one in a fraction of a second when needed. The clever part: when a batch of customers arrives, you run the big press once for everyone, then walk down the line applying each person's own stamp to just their page. Nobody waits for a separate press run. One press plus a drawer of cheap stamps serves a whole city of customers.

Key concepts

LoRA adapters S-LoRA serving Paged adapter memory Segmented batched matmul

Quick scan · 2–4 min

Everything important, quickly.

4 min: why merging fails for multi-tenant + base resident plus paged adapters + segmented batched matmul (BGMV/SGMV) over heterogeneous rows + CPU paging for cold adapters + production economics.

Side-by-side

Strategy	GPU memory cost	Batch behavior	Tenants per GPU
Merge adapter into base	One full model per tenant	One model per request	Handful
Swap single active adapter	Base plus one adapter	Serialized, one adapter at a time	Many but slow
S-LoRA paged plus segmented kernel	Base plus paged adapter pool	One batched pass, heterogeneous adapters	Thousands

Real-world examples

Real products, models, and research that use this idea.

S-LoRA demonstrated serving roughly 2000 adapters on a single A100 by paging adapters and running unified batched kernels over the base model.
Punica introduced the SGMV kernel that batches forward passes across distinct LoRA adapters, the kernel primitive S-LoRA builds on.
vLLM ships multi-LoRA serving for Llama 4 and Qwen 3, letting one deployment hot-swap hundreds of customer adapters per request.
Predibase LoRAX serves many fine-tuned adapters over one base model for multi-tenant production fine-tuning at low cost per tenant.
NVIDIA TensorRT-LLM supports multi-LoRA inference with batched heterogeneous adapters on H100 and B200 serving stacks.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy can a standard batched matmul not handle a batch where each row uses a different adapter?

Sign in or go Pro to see the approach for this follow-up.

QWhat is the difference between BGMV and SGMV, and when does each run?

Sign in or go Pro to see the approach for this follow-up.

QHow does paged adapter storage handle adapters of different ranks without fragmentation?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.