Shared experts add stable baseline FFN capacity and ease router learning, but raise fixed per-token compute, they do not fix load balance or guarantee uniform routing.
Adding a shared expert is like hiring a full-time prep cook while keeping rotating specialists. Meals get more consistent, specialists focus on their niche, but you always pay the prep cook's salary even on slow nights.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Multi-select MoE questions reward precise tradeoff thinking. Shared experts are not a universal upgrade, they are a deliberate exchange of sparsity for baseline reliability. Candidates who only see the benefits pick A and B and miss C; candidates who want a silver bullet pick D or E and overstate the fix.
This deep dive maps each option to mechanism-level reasoning so you can defend your selections under follow-up pressure.
The meta-skill: every MoE architectural knob has a benefit, a cost, and a list of problems it does not solve.
Multi-select MoE questions punish both under-selection (missing real tradeoffs) and over-selection (treating shared experts as a silver bullet). The correct set is A, B, C, three true statements about benefits and costs. D and E are traps for candidates who want shared experts to solve every MoE problem. This deep dive maps each option to mechanism-level reasoning.
Senior interviewers use this multi_select to see whether you treat shared experts as a free lunch. The correct answer set is deliberately incomplete without C, benefits without cost is the most common partial-credit failure.
Senior interviewers reward candidates who connect this mechanism to a concrete deployment or training decision, not only the textbook definition. Close with one number, one failure mode, and one monitoring signal you would track in production.
Why A and B are true, the quality and router story
Baseline capacity (A). When routed experts collapse, a few experts receive most tokens, tokens routed to under-trained experts get weak transforms. Shared FFN paths guarantee a minimum-quality nonlinear expansion on every token regardless of routing lottery.
Router relief (B). The gate network otherwise learns both specialization (which expert for which pattern) and universality (features every token needs). Splitting universality into shared experts simplifies the routed router's job, it can focus on heterogeneous specialization.
Quality mechanism for A. When routed experts collapse, tokens assigned to under-trained experts receive weak FFN transforms. Quality degrades unevenly across the input distribution, code tokens routed to a weak expert suffer more than chitchat tokens routed to a strong one. Shared FFN paths guarantee minimum-quality nonlinear expansion on every token regardless of routing lottery outcome. For B, the router otherwise learns both specialization routing and universal feature preservation, a harder dual objective that slows convergence and increases collapse risk early in training.
Real training logs often show quality variance across routed experts before shared paths stabilize. Teams adding shared experts mid-architecture frequently report smoother loss curves in early epochs, not because routed experts improved, but because the dense floor absorbed generic transforms.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepSeek-V2/V3 added shared experts while retaining routed-expert balancing mechanisms.
- Mixtral's routed-only design relies more heavily on aux loss and top-2 redundancy instead of shared paths.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you A/B test whether shared experts justify their compute cost?
Compare quality metrics and routed-expert CV with/without shared path at matched active FLOP budget.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Selecting the options claiming shared experts eliminate aux loss or guarantee uniform utilization, those overstate the fix.
60 second bullets to scan on the way to the call.
Quality stability from dense baseline
Router burden reduction for universal features
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.