Offloading experts to CPU RAM makes decoding PCIe bound. Routing changes every token and layer, so weight blocks cross a link roughly 100x slower than HBM. Latency varies with cache luck.
Imagine a chef whose kitchen counter only fits a few ingredient bins, so the rest live in a basement pantry. Every dish needs a different combination of ingredients, and orders arrive one dish at a time. When an ingredient is already on the counter, cooking is instant. When it is not, the chef stands around waiting for a slow dumbwaiter to haul a heavy bin upstairs. Some dishes get lucky and use what is already out. Others stall for the dumbwaiter three times in a row. The chef's chopping speed never changed, only the waiting did. That is expert offloading: the GPU computes as fast as ever, but cold experts ride a slow elevator up from CPU memory, and every token has different luck.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Expert offloading sounds like it has no cost. An MoE activates only a couple of experts per token, so why keep all of them in scarce GPU memory? Keep the cold ones in cheap CPU RAM and stream them in when the router asks. The model still does the same math, so surely latency barely moves.
The real cost is the memory traffic that conditional compute creates, which FLOPs accounting hides. Dense models read the same weights every token, which makes their memory traffic predictable and cacheable. MoE models read different weights every token and every layer. So offloading converts a compute pipeline into a storage system with a very high miss penalty.
Systems reasoning settles it. Identify the slowest link the data must cross, and count how often it must be crossed. Then notice that the count changes token by token. Seen that way, option A is almost forced. The other three options each fail on a single concrete fact.
The bandwidth gap between HBM and PCIe
Start with the numbers, because they decide everything. An H100's HBM3 delivers roughly 3 TB/s to the compute units. PCIe Gen4 x16, the link between CPU RAM and the GPU, delivers about 32 GB/s in practice. That is roughly a 100x drop. Any byte that must cross PCIe at decode time is a byte read 100x slower than a resident byte.
Now size the data to move. A Mixtral 8x7B expert is a SwiGLU FFN with three matrices of 4096 × 14336 weights, about 176M parameters, or roughly 350 MB in fp16. The transfer time for one cold expert is:
That is eleven milliseconds for one expert, at one layer, for one token. The matmul that expert performs on a single token takes well under a millisecond. The transfer is not an overhead on the compute. It exceeds the compute by orders of magnitude.
Compare the dense baseline. A dense model with the same active parameter count keeps everything resident and reads it from HBM at full speed every token. Offloading does not change what the MoE computes. It changes where the weights live, and at decode time, where weights live is what sets latency.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Path | Approximate bandwidth | Implication for offloading |
|---|---|---|
| GPU HBM3 | About 3 TB/s | Resident experts read at full speed |
| NVLink, GPU to GPU | Hundreds of GB/s | Sharding experts across GPUs beats host offload |
| PCIe Gen4 x16, CPU to GPU | About 32 GB/s | Each cold expert miss costs milliseconds |
| CPU DRAM, compute in place | About 100 GB/s | Often faster to run the expert on CPU than to move it |
Real products, models, and research that use this idea.
- The Mixtral offloading work from late 2023 combined an LRU expert cache with speculative prefetch. It still landed near 2 to 3 tokens per second on consumer GPUs, an order of magnitude below fully resident serving.
- KTransformers runs DeepSeek-R1 and Kimi K2 on workstations by keeping attention on the GPU and computing cold experts on the CPU. It moves small activations instead of huge weight blocks.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you design a speculative prefetcher for offloaded experts?
Hidden states going into layer i+1 are partly predictable from layer i. Score candidate experts early and start copies while attention computes.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming identical active FLOPs means identical latency. With offloading, the bottleneck moves from compute to PCIe weight transfers, and routing variance makes those transfers unpredictable.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.