Switch top-1 minimizes expert FFN compute per token; Mixtral top-2 doubles it for better quality and routing redundancy.
Imagine calling one doctor per visit versus two. One call is cheaper and faster, but if that doctor is swamped, quality suffers. Two calls cost more but you get a second opinion and a backup if the first is overloaded, that is the Switch top-1 vs Mixtral top-2 tradeoff.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Top-1 versus top-2 is the MoE equivalent of "how many specialists do you consult per decision?" Switch and Mixtral anchor opposite points on that spectrum, and interviewers expect you to name both the landmark model and the tradeoff in one breath.
This flashcard is deliberately concise: one sentence per k. The deep dive unpacks compute, quality, collapse, and serving implications so follow-up questions do not catch you flat.
Master this contrast before higher-k frontier models, the same tradeoff logic scales.
Switch and Mixtral anchor opposite points on the k spectrum, the two landmark models every MoE interviewer expects you to name. This flashcard is deliberately concise: one sentence per k. The deep dive unpacks compute, quality, collapse, and serving implications so follow-up questions on dispatch fanout and gradient flow do not catch you flat.
The k parameter is the single dial that moves FFN compute, dispatch communication, collapse risk, and quality redundancy in tandem. Naming Switch and Mixtral as anchors gives interviewers confidence you understand the dial, not just the current default.
Memorize the two one-liners: Switch top-1 minimizes expert FFN compute with higher collapse risk; Mixtral top-2 doubles expert FFN cost for quality and routing redundancy. Everything else in this deep dive elaborates those two sentences.
Senior interviewers reward candidates who connect this mechanism to a concrete deployment or training decision, not only the textbook definition. Close with one number, one failure mode, and one monitoring signal you would track in production.
What k controls in the MoE FFN sublayer
Active expert paths. Given N experts and router output, top-k selects k expert FFN modules to run on each token. Non-selected experts contribute zero FFN FLOPs for that token (parameters still reside in memory).
Expert FFN cost per token scales approximately linearly with k:
Attention and other dense layers are unchanged. k is the sparse FFN dial.
What k does not change. Attention sublayers, embedding layers, layer norms, and the router itself are unaffected by k. Only the sparse FFN sublayer scales with k. When comparing Switch versus Mixtral total model FLOPs, the expert FFN delta is the primary driver, attention and other dense components are shared.
Router cost is identical for top-1 and top-2, same W_router matmul, same softmax. The k difference is entirely in how many expert FFNs execute after selection.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Switch Transformer (Google, 2021) is the canonical top-1 + capacity factor reference.
- Mixtral 8x7B (Mistral, 2023) is the canonical open top-2 MoE product model.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy did Switch add capacity factor alongside top-1?
Expert buffers overflow and drop tokens, capacity factor limits per-expert batch load.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forgetting the compute doubling tradeoff or attributing top-1 to Mixtral and top-2 to Switch.
60 second bullets to scan on the way to the call.
k=1 vs k=2 expert FFN matmuls per token
Switch as top-1 landmark
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.