What role do shared (always on) experts play in DeepSeek style MoE stacks?
Shared experts are always-on FFN modules that give every token a dense baseline transform while routed experts add conditional specialization.
Imagine a restaurant with specialty chefs who only cook certain dishes when ordered, plus one head chef who touches every plate. Routed MoE experts are the specialists, they fire only when the router picks them. Shared experts are the head chef, they run on every token so basic seasoning never depends on whether routing worked well that batch.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Shared experts sound like a minor architectural tweak, but they encode a real design judgment: pure sparse routing asks the gate network to do two jobs at once, pick specialists and preserve universal FFN behavior. Production MoE stacks that skip this lesson often discover quality cliffs when a handful of routed experts hog traffic.
DeepSeek and several follow-on open models answered that problem by adding always-on FFN paths beside the routed bank. This deep dive explains what those modules actually do, why they help, and what you give up in return.
If you can explain shared experts as a dense baseline plus conditional specialists, with a concrete cost line, you are already ahead of most MoE interview answers that stop at naming DeepSeek.
The flashcard tests a precise architectural distinction that many candidates blur: shared experts are not popular routed experts and not GPU replication for parallelism. They are a separate class of always-on FFN modules with a specific design motivation. Understanding that distinction is increasingly important as DeepSeek-style architectures become the reference point for open-weight MoE in 2026.
Senior interviewers reward candidates who connect this mechanism to a concrete deployment or training decision, not only the textbook definition. Close with one number, one failure mode, and one monitoring signal you would track in production.
What shared experts are in the block recipe
The mechanism. A standard MoE FFN sublayer has a router and N expert FFNs. For each token, the router scores experts, picks top-k, runs only those k FFNs, and combines outputs with gate weights.
Shared experts are additional FFN modules whose forward pass is unconditional: every token runs them, every step, independent of routing logits. Their output is typically added to the routed-expert sum before the residual connection.
Formally, for token hidden state h:
The shared term is never zeroed by routing. That is the entire point, baseline capacity is guaranteed.
Block-level placement. In a DeepSeek-style MoE block, the forward pass computes routed expert outputs first (top-k selection, weighted sum), then adds shared expert outputs unconditionally, then applies the residual connection. The shared path is additive, not a replacement for routing. Multiple shared experts are possible, DeepSeek-V2 uses a small number of shared FFN modules alongside a large routed bank of fine-grained experts.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepSeek-V2 and DeepSeek-V3 use shared experts alongside fine-grained routed experts in their open MoE stacks.
- Several 2025–2026 open MoE releases follow the shared plus routed pattern for training stability.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow many shared experts do frontier stacks typically use, and why not make all experts shared?
Contrast one to two shared FFNs vs full dense width; name the sparsity budget tradeoff.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating shared experts as redundant copies of routed experts rather than an unconditional dense baseline.
60 second bullets to scan on the way to the call.
Shared vs routed activation rule
Why universal features need a dense path
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.