Predict the outcome when routed tokens exceed an expert's capacity buffer
A Switch style MoE layer has N=4 experts, capacity factor C=1.0, and a training batch of 32 tokens. Expert 2 receives 12 routed tokens, but its buffer holds only C × (32/4) = 8 tokens. What happens to the 4 overflow tokens?
Overflow tokens are dropped or skip FFN via residual, Switch-style capacity buffers cap per-expert tokens at C × (batch/N).
Imagine each expert has a fixed number of seats per train ride (capacity buffer). If 12 passengers want expert 2 but only 8 seats exist, 4 passengers wait at the platform, they either skip the ride (dropped) or continue without the expert's processing (residual skip). The train does not stall.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Predicting what happens when routed tokens exceed an expert's capacity buffer tests whether you understand Switch Transformer's training-time dispatch mechanics. The scenario is concrete: 32 tokens, 4 experts, capacity factor 1.0, expert 2 gets 12 tokens against an 8-token buffer. Four tokens overflow.
Capacity factor is one of MoE's most important systems concepts that pure architecture discussions miss. It exists because expert-parallel training shards experts across GPUs, and each GPU must process a bounded number of tokens per step for predictable performance. Without capacity limits, a routing hot-spot could send unbounded tokens to one expert, stalling the entire training step.
This deep dive explains the capacity formula, overflow handling, and the trade-offs in choosing C.
The sections below build from intuition to production practice. Read actively: after each section, pause and restate the key point in your own words, that rehearsal is what converts reading into interview-ready recall.
The capacity buffer formula
Per-expert buffer size in Switch-style training:
where C is capacity factor, T is total tokens in the batch step, and N is number of experts. With C=1.0, T=32, N=4: capacity = 1.0 × 32/4 = 8 tokens per expert.
C=1.0 means the buffer fits exactly the expected load under uniform routing (each expert gets T/N tokens). C>1.0 adds headroom for routing skew; C<1.0 forces tighter bounds with more drops.
The buffer is a per-step, per-expert quota enforced during expert-parallel dispatch, not a memory allocation concept.
Track drop rate as a training metric: sustained drop rates above 1–2% suggest C is too tight or routing is too skewed. Zero drops with low C may indicate underutilized buffer capacity.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Switch Transformer introduced capacity factor with explicit token-drop overflow handling.
- GShard training uses capacity buffers for expert-parallel load management.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would C=1.5 change the buffer size and drop rate?
Buffer = 1.5 × 8 = 12 per expert; expert 2's 12 tokens fit exactly, zero drops for this case.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming overflow tokens queue until the expert finishes or block the training step.
60 second bullets to scan on the way to the call.
Capacity buffer formula C × T/N
Overflow = drop or residual skip
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.