Predict per expert capacity for 4096 tokens with 8 experts, top-2 routing, capacity factor 1.25.
A Switch style MoE layer processes a batch of 4096 tokens with 8 experts, top-2 routing, and capacity factor 1.25. Expert capacity is computed as capacity_factor x (tokens x top_k) / num_experts. How many token slots does each expert get?
Capacity = C x (T x k) / N. Here 1.25 x (4096 x 2) / 8 = 1280 slots per expert: the balanced share of 1024 assignments plus 25% headroom for routing skew.
Picture a theater with 8 ticket counters, where every visitor must stop at 2 of them. If 4096 visitors arrive, the counters handle 8192 stops in total, and a fair queue gives each counter 1024 stops. Now the manager knows queues are never perfectly fair: some counters are just more popular. So she puts 25% more chairs at every counter than the fair share, 1280 chairs each, as insurance. That insurance number is the expert's capacity. If more than 1280 visitors show up at one counter, the extras are politely waved through without service. The MoE layer works the same way: the capacity factor of 1.25 is the manager's margin for popularity, and overflow tokens skip the expert rather than making everyone wait.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Capacity factor questions look like arithmetic quizzes, but they are really systems questions wearing a math costume. The formula being tested, capacity = C x (T x k) / N, encodes one of the central compromises in MoE training: routers are free to send tokens anywhere, yet hardware wants every expert's workload bounded and known before the step begins.
The scenario gives concrete values: 4096 tokens, 8 experts, top-2 routing, capacity factor 1.25. Getting 1280 requires respecting the order of operations, and the most common wrong answers each correspond to a real conceptual gap, not a slipped digit.
This deep dive works the calculation carefully, then spends most of its time on what the number means: why buffers exist at all, what happens to tokens that overflow them, how teams tune C in practice, and why a newer family of dropless kernels makes the whole formula optional. That context is what turns a correct answer into a convincing one.
Three factors, strict order
The formula composes three ideas, and each has a physical meaning.
Start with total assignments. Under top-k routing, every one of the T tokens is dispatched to k distinct experts, so the layer's total workload is T x k assignments. Here that is 4096 x 2 = 8192. Skipping this step is the classic error: dividing 4096 by 8 first gives 512, which is the answer for a top-1 layer, not this one.
Next, the balanced share. If routing were perfectly uniform, each of the N = 8 experts would receive 8192 / 8 = 1024 assignments. This is the same k/N reasoning that governs utilization baselines, scaled to absolute counts.
Finally, headroom:
The capacity factor inflates the fair share by 25% because real routers skew. Some experts are simply more popular for a given batch, and a buffer sized to the exact average would overflow on every step where the router is anything short of perfect. C is insurance, priced in slots.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Switch Transformer introduced this exact capacity formula and reported tuning C between 1.0 and 1.25 for trillion-parameter training.
- GShard used capacity buffers with fixed dispatch tensor shapes to keep expert-parallel TPU training compilable and predictable.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat changes if this layer switches to top-1 routing with everything else fixed?
Recompute total assignments; the balanced share halves, so the same C yields half the slots per expert.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Computing 4096 / 8 = 512 and scaling that, which forgets top-2 routing doubles total assignments; the balanced base is 1024 per expert, not 512.
60 second bullets to scan on the way to the call.
The capacity formula and the order of its three factors
Why total assignments are tokens times k
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.