Given 4 experts of 2B params each, top-1 routing, and 5B always on shared layers, how many active parameters does one token use?
A MoE transformer has shared attention, embedding, and output layers totaling 5B parameters, these run on every token. One MoE FFN layer has N=4 experts, each expert FFN is 2B parameters. Top-k routing with k=1 selects one expert per token. Router weights are negligible. How many billion parameters are actively used per token (round to a whole number)?
Active params per token = always-on shared layers (5B) + k active experts (1 × 2B) = 7B; total stored would be 13B.
Think of a building with a front desk and four specialist offices. Every visitor checks in at the front desk (shared layers, 5B). Then a receptionist sends each visitor to exactly one specialist office (k=1, one 2B expert). The visitor's journey uses the desk plus one office, 7B total, even though four offices exist in the building (13B if you counted every office).
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Active versus total parameter counting is the arithmetic backbone of every MoE interview question. The scenario here is deliberately minimal: one MoE FFN layer, four experts, top-1 routing, and a 5B shared stack. The math takes thirty seconds once you know the template, but the conceptual mistake (reporting total when asked for active) appears constantly in real interviews and in production mis-sizing.
MoE's core promise is decoupling capacity from compute. You store N expert FFNs (high capacity, many specialists) but execute only k of them per token (low compute, sparse activation). The shared stack, embeddings, attention, output layers, runs densely on every token and must always appear in the active count.
This deep dive builds the formula, walks through the toy scenario, and extends to multi-layer stacks and production reporting conventions.
The sections below build from intuition to production practice. Read actively: after each section, pause and restate the key point in your own words, that rehearsal is what converts reading into interview-ready recall.
The active vs total formula
For a single MoE FFN layer with N experts of size E each, top-k routing, and shared layers S:
Router weights are typically small (a linear projection from hidden dim to N logits) and often omitted in back of envelope estimates. The scenario gives S=5B, N=4, E=2B, k=1: active = 5 + 1×2 = 7B.
Total = 5 + 4×2 = 13B. The gap between 7B and 13B is exactly the sparsity benefit: you get four specialists' worth of capacity while paying one specialist's FFN compute per token.
In multi-layer stacks, repeat the per-layer expert contribution for each MoE FFN layer while counting shared layers once. Interviewers sometimes extend the toy scenario, ask whether they mean one layer or the full model before multiplying.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Switch Transformer uses top-1 routing, active FFN compute is one expert per token per MoE layer.
- Capacity planning for MoE inference separates GPU memory (total) from throughput (active FLOPs).
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does active count change if the model has 32 MoE layers with the same config?
Shared layers counted once; each MoE layer adds k × 2B active and N × 2B total expert params.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reporting 13B (total params) or 8B (using k=2 by mistake) instead of 5B + 1×2B = 7B active.
60 second bullets to scan on the way to the call.
Active vs total parameter formula
Effect of k on active compute
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.