Forecast the latency profile when cold experts live in CPU RAM and stream in on demand.
Offloading experts to CPU RAM makes decoding PCIe bound: routing changes every token and layer, so weight blocks cross a link roughly 100x slower than HBM, and latency varies with cache luck.
Imagine a chef whose kitchen counter only fits a few ingredient bins, so the rest live in a basement pantry. Every dish needs a different combination of ingredients, and orders arrive one dish at a time. When an ingredient is already on the counter, cooking is instant. When it is not, the chef stands around waiting for a slow dumbwaiter to haul a heavy bin upstairs. Some dishes get lucky and use what is already out; others stall for the dumbwaiter three times in a row. The chef's chopping speed never changed, only the waiting did. That is expert offloading: the GPU computes as fast as ever, but cold experts ride a slow elevator up from CPU memory, and every token has different luck.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Expert offloading sounds like a free lunch. An MoE activates only a couple of experts per token, so why keep all of them in scarce GPU memory? Park the cold ones in cheap CPU RAM and stream them in when the router asks. The model still does the same math, so surely latency barely moves.
This question tests whether you can see past FLOPs accounting to the memory traffic that conditional compute creates. Dense models read the same weights every token, which makes their memory traffic predictable and cacheable. MoE models read different weights every token and every layer, which means offloading converts a compute pipeline into a storage system with a brutal miss penalty.
The skill being probed is systems reasoning: identify the slowest link the data must cross, count how often it must be crossed, and notice that the answer changes token by token. Once you frame it that way, option A is almost forced, and the other three options each collapse on a single concrete fact.
The bandwidth cliff between HBM and PCIe
Start with the numbers, because they decide everything. An H100's HBM3 delivers roughly 3 TB/s to the compute units. PCIe Gen4 x16, the link between CPU RAM and the GPU, delivers about 32 GB/s in practice. That is roughly a 100x cliff. Any byte that must cross PCIe at decode time is a byte read 100x slower than a resident byte.
Now size the cargo. A Mixtral 8x7B expert is a SwiGLU FFN with three matrices of 4096 × 14336 weights, about 176M parameters, or roughly 350 MB in fp16. The transfer time for one cold expert is:
Eleven milliseconds, for one expert, at one layer, for one token. The matmul that expert performs on a single token takes well under a millisecond. The transfer is not an overhead on the compute; it dwarfs the compute by orders of magnitude.
Compare the dense baseline: a dense model with the same active parameter count keeps everything resident and reads it from HBM at full speed every token. Offloading does not change what the MoE computes. It changes where the weights live, and at decode time, where weights live is the whole game.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Path | Approximate bandwidth | Implication for offloading |
|---|---|---|
| GPU HBM3 | About 3 TB/s | Resident experts read at full speed |
| NVLink, GPU to GPU | Hundreds of GB/s | Sharding experts across GPUs beats host offload |
| PCIe Gen4 x16, CPU to GPU | About 32 GB/s | Each cold expert miss costs milliseconds |
| CPU DRAM, compute in place | About 100 GB/s | Often faster to run the expert on CPU than to move it |
Real products, models, and research that use this idea.
- The Mixtral offloading work from late 2023 combined an LRU expert cache with speculative prefetch and still landed near 2 to 3 tokens per second on consumer GPUs, an order of magnitude below fully resident serving.
- KTransformers runs DeepSeek-R1 and Kimi K2 on workstations by keeping attention on the GPU and computing cold experts on the CPU, moving small activations instead of huge weight blocks.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you design a speculative prefetcher for offloaded experts?
Hidden states going into layer i+1 are partly predictable from layer i; score candidate experts early and start copies while attention computes.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming identical active FLOPs means identical latency. With offloading, the bottleneck moves from compute to PCIe weight transfers, and routing variance makes those transfers unpredictable.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.