Identify the memory bottleneck that emerges when your MoE server moves to 128k-token contexts.
MoE sparsifies the FFN but leaves attention dense, so KV cache scales like any dense model and at 128k tokens it starts competing with the expert weights for VRAM.
Think of an MoE model as a building with many specialist rooms upstairs and one big shared lobby downstairs. The MoE trick only shrinks the upstairs: fewer specialists are working at any moment. The lobby is still the same and every visitor has to remember every conversation that happened with them. When visits get very long, the lobby fills with memos. Eventually those memos take up more space than the specialists upstairs. The bottleneck has moved from the upstairs (compute) to the lobby (memory). Fixing it means making memos smaller, not making the specialists work less.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Long-context MoE models look paradoxical at first glance. They advertise sparsity, which sounds like it should help with memory. They handle 128k tokens, which sounds like it should require memory. Both can be true because the sparsity and the long context affect entirely different parts of the architecture.
This deep dive locates the bottleneck precisely, contrasts it with the seductive but wrong alternatives, and walks through the attention-side techniques that long-context MoEs actually rely on.
Where MoE changes the architecture and where it does not
A standard transformer layer has two sublayers: multi-head attention and a feed-forward network. MoE replaces the FFN sublayer with a sparse mixture of expert FFNs plus a router. Top-k of those experts run per token. Everything else, including the entire attention sublayer, is untouched.
That structural fact has two memory consequences. First, expert weights for every expert must be resident in VRAM, because the router can select any of them next. Total parameters set the weight footprint. Second, the attention sublayer behaves exactly like a dense model's. The KV cache, the QKV projection matrices, and the attention output projection are all standard transformer parts.
When people say MoE is sparse they mean FLOPs per token are reduced. Per-step FFN compute scales with active parameters, not total. But memory does not work that way. Weight memory is set by total parameters, and attention memory is set by the dense attention sublayer.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepSeek-V3 uses multi-head latent attention to compress KV state, key to its 128k context economics
- Mixtral 8x22B pairs MoE with GQA so 32k contexts stay viable on standard 8xH100 nodes
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does multi-head latent attention reduce KV memory and what does it trade?
MLA projects K and V into a low-rank latent space then reconstructs per head on the fly. It trades a small amount of attention compute and code complexity for a large reduction in stored bytes per token.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming the MoE structure changes attention memory or that routing decisions accumulate in cache as sequence length grows.
60 second bullets to scan on the way to the call.
Which transformer sublayer the MoE replaces and which it does not
How the KV cache formula scales in layers, KV heads, head dim, context, and batch
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.