Drag each answer to line up with its matching prompt
Many small expert FFNs instead of a few large ones
Auxiliary loss free balancing
Always on FFN modules that run on every token alongside routed experts
Token choice routing
Per expert bias terms adjusted from recent load outside backprop
Shared experts
Each token independently selects its top-k experts via router softmax
Fine-grained experts
DeepSeek MoE combines fine-grained routed experts, always-on shared FFNs, token-choice top-k routing, and aux loss free bias balancing, a distinct recipe from Mixtral-style 8×7B stacks.
Think of DeepSeek's MoE like a hospital with many small specialist rooms plus a general ward that every patient visits first. Fine-grained experts are the many small rooms, more specialists, each handling a narrower slice. Shared experts are the general ward that runs on every token no matter what. Token-choice routing means each patient (token) picks their own specialists. Aux-loss-free balancing is the scheduling desk that nudges traffic when one specialist gets overloaded, without adding a separate penalty to the main treatment goal the way Switch Transformer aux loss does.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
DeepSeek's MoE architecture became a landmark interview topic because it packages four distinct design choices that together challenge the Mixtral mental model most candidates carry. Interviewers are not asking you to recite parameter counts, they want to know whether you can map each mechanism to the problem it solves.
The four pairs in this match question, fine-grained experts, shared experts, token-choice routing, and auxiliary loss free balancing, each address a different axis: capacity granularity, routing insurance, inference compatibility, and load stability. Understanding them separately is what separates a surface-level 'DeepSeek is MoE' answer from a production-aware one.
This deep dive walks through each choice, contrasts it with the Mixtral/Switch lineage, and closes with how you would explain the stack in a senior ML systems interview.
Picture yourself in a system design round where the interviewer draws two boxes on the whiteboard, Mixtral on the left, DeepSeek on the right, and asks you to label what is different inside each MoE block. The match pairs in this question are exactly those labels. If you can explain why each choice exists, you can also defend GPU sizing, balancing policy, and serving topology decisions downstream.
Fine-grained experts: more specialists, smaller footprints
The mechanism. Standard MoE replaces one dense FFN with N expert FFNs plus a router. Mixtral's famous 8x7B config uses eight large experts, each roughly a 7B-class FFN, with top-2 active per token. DeepSeek-V2/V3 push in the opposite granularity direction: many more experts, each smaller.
Why bother? With more routing slots, the gating network can assign narrower functional niches. Overload risk on any single expert drops when traffic spreads across a larger pool. Total expert-parameter budget can scale up while active FLOPs per token stay bounded by top-k.
The systems cost is real. More experts means more shards in expert-parallel layouts, more all to all dispatch groups, and more utilization metrics to monitor. Fine-grained routing is a bet that specialization and load-spreading gains outweigh communication and memory-management overhead at frontier scale.
A useful sizing exercise: DeepSeek-V3 reports hundreds of routed experts with a smaller per-expert FFN than Mixtral's eight large slots. That means the router has more bins to spread traffic across, but your serving stack must shard and load more expert weight tensors. Fine-grained is not a free quality knob, it is a bet that overload risk drops faster than dispatch complexity rises.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepSeek-V2 and DeepSeek-V3 open-weight models use fine-grained routed experts plus shared always-on FFNs with aux loss free bias balancing.
- Mistral's Mixtral 8x7B uses eight large routed experts with top-2 activation and Switch-style auxiliary load balancing, a contrasting landmark config.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy add shared experts if routed experts already provide capacity?
Argue baseline dense FFN insurance: routing can misfire or collapse partially; shared FFNs guarantee every token gets a stable transformation independent of top-k picks.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing shared experts with routed experts, or assuming DeepSeek uses the same Switch-style auxiliary load-balancing loss as Mixtral.
60 second bullets to scan on the way to the call.
Fine-grained vs coarse expert granularity
Role of shared always-on FFNs
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.