Shared experts add stable baseline FFN capacity and ease router learning, but raise fixed per-token compute. They do not fix load balance or guarantee uniform routing.
Adding a shared expert is like hiring a full-time prep cook while keeping rotating specialists. Meals get more consistent, specialists focus on their niche, but you always pay the prep cook's salary even on slow nights.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Shared experts are not a universal upgrade. They are a deliberate exchange of sparsity for baseline reliability. Options A, B and C are three true statements about benefits and costs. Picking only A and B misses the cost in C. Picking D or E overstates the fix by treating shared experts as a solution to every routing problem.
Every MoE architectural knob has a benefit, a cost, and a list of problems it does not solve.
Why A and B are true: the quality and router story
Baseline capacity (A). When routed experts collapse, a few experts receive most tokens, and tokens routed to under-trained experts get weak transforms. Quality degrades unevenly across the input distribution: code tokens routed to a weak expert suffer more than chitchat tokens routed to a strong one. Shared FFN paths guarantee a minimum-quality nonlinear expansion on every token, regardless of how routing turns out.
Router relief (B). The gate network otherwise learns both specialization (which expert for which pattern) and universality (features every token needs). That harder dual objective slows convergence and increases collapse risk early in training. Splitting universality into shared experts simplifies the router's job, so it can focus on heterogeneous specialization.
Real training logs often show quality variance across routed experts before shared paths stabilize. Teams adding shared experts mid-architecture frequently report smoother loss curves in early epochs. The reason is that the dense floor absorbed generic transforms, not that routed experts improved.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepSeek-V2/V3 added shared experts while retaining routed-expert balancing mechanisms.
- Mixtral's routed-only design relies on top-2 redundancy instead of shared paths.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you A/B test whether shared experts justify their compute cost?
Compare quality metrics and routed-expert CV with/without shared path at matched active FLOP budget.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Selecting the options that claim shared experts eliminate aux loss or guarantee uniform utilization overstates the fix.
60 second bullets to scan on the way to the call.
Quality stability from dense baseline
Router burden reduction for universal features
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.