Order the historical MoE milestones chronologically
- 1Switch Transformer simplifies to top-1 routing with capacity factor
- 2GShard demonstrates trillion token MoE training with top-2 routing
- 3Mixtral releases quality top-2 MoE with open weights
MoE lineage runs GShard (2020, top-2 at scale) → Switch Transformer (2021, top-1 simplification) → Mixtral (2023, open top-2 quality).
Think of MoE history like phone evolution: first a lab prototype proves the idea works at huge scale (GShard), then a simpler cheaper model hits mass production (Switch top-1), then a popular open product makes everyone want one (Mixtral). Each step built on the last.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
MoE interview questions often jump straight to Mixtral naming conventions without the research history that made Mixtral possible. Ordering GShard, Switch, and Mixtral tests whether you know why top-k evolved, not just what k Mixtral uses today.
Each milestone answered a different bottleneck: can we train this at all (GShard), can we simplify dispatch (Switch), can we ship it openly at quality (Mixtral).
This deep dive gives you dates, routing choices, and the design lesson each paper contributed, enough to defend the order and handle follow-ups about top-1 vs top-2.
Interview recitation tip: name the three papers in order with one anchor each, GShard (scale plus capacity), Switch (top-1 plus aux loss), Mixtral (open top-2). Then note what changed between steps: expert count, routing k, balancing mechanism. DeepSeek-V3 extends the same arc with shared experts and bias balancing instead of classical aux loss.
GShard (2020): proving scale with top-2
The scale proof. GShard (Lepikhin et al., Google, 2020) showed conditional computation could train models toward trillion-parameter scale using expert parallelism and top-2 routing, each token activated two experts.
Before GShard, MoE was academically known (Shazeer 2017) but not proven at LLM training scale with modern hardware sharding. GShard's legacy is systems + scale: automatic sharding, all to all dispatch, and demonstration that sparse FFN layers could replace dense layers in massive transformers.
Interview anchor: GShard = top-2 + trillion-scale proof.
Pre-GShard context. Before GShard, MoE existed academically (Shazeer 2017 sparsely-gated layers in LSTM stacks) but had not been proven at LLM training scale with modern hardware sharding. GShard's contribution was systems plus scale: automatic sharding, all to all dispatch, and demonstration that sparse FFN layers could replace dense layers in massive transformers. Top-2 routing was the default because quality sensitivity to top-1 was already understood from earlier experiments.
GShard also introduced conditional computation at scale in multilingual translation models before the technique migrated to general LLM pretraining. The trillion-parameter proof of concept changed what labs believed was trainable.
Interview follow-up. When asked why GShard used top-2, answer quality sensitivity: early MoE training was unstable at top-1 without capacity mechanisms. GShard prioritized proving scale before optimizing per-token FLOPs.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Google's GShard (2020) and Switch Transformer (2021) papers define the research lineage most MoE interviews reference.
- Mistral Mixtral 8x7B (December 2023) sparked widespread open MoE adoption and naming conventions like 8x7B.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat problem did Switch's top-1 routing solve that GShard's top-2 did not?
Name dispatch simplicity, lower active expert FLOPs, and capacity factor handling, with quality tradeoff.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Placing Mixtral before Switch because Mixtral is more famous in open-source circles today.
60 second bullets to scan on the way to the call.
GShard 2020 top-2 at trillion scale
Switch 2021 top-1 plus capacity factor
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.