Why do DeepSeek style stacks slice the FFN into many small experts instead of a few large ones?
Many small experts with higher top-k expose far more expert combinations per token, letting knowledge decompose more cleanly at the same active compute. The cost is more routing and dispatch bookkeeping.
Picture two kitchens with the same ingredients budget. One has 8 chefs who each cover broad cuisines, and a customer's meal gets routed to 2 of them. The other has 64 chefs, each specialised in a narrow style, and each meal gets routed to 6 of them. The first kitchen can combine its chefs in 28 ways. The second can combine its chefs in billions of ways. With more, narrower specialists, the kitchen can match exactly the right blend of skills to each dish, even though both kitchens use the same total chef-hours per meal. The trade is more menu decisions and a busier dispatcher.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Fine-grained expert segmentation is one of the design choices that distinguishes DeepSeek-V2 and V3 from the Mixtral lineage. On its face it sounds like a minor tweak: instead of eight large experts with top-2 routing, use sixty-four small experts with top-6. The total parameter count and the active parameter count are held constant. Why bother?
The answer is combinatorial. At the same arithmetic budget per token, a fine-grained design exposes the router to a vastly larger combinatorial space of expert blends. Knowledge can decompose into smaller, more clearly specialised pieces, and the router learns to compose them on demand. This walkthrough unpacks the argument, names what does and does not grow with segmentation, and explains when the trade is worth taking.
Mental model: experts are puzzle pieces. Few large pieces force each piece to cover a broad area imperfectly. Many small pieces let the model compose exactly the right shape for each token.
The combinatorial argument
For N experts and top-k routing, the router picks an unordered subset of size k from N experts, giving a combinatorial space of size N choose k. For 8 experts top-2 that is 28 possible expert pairs. For 64 experts top-6 that is roughly 74 million possible six-expert subsets.
This matters because the model's per-token computation is determined by which experts run. A larger combinatorial space gives the router more freedom to assemble the right blend of specialisations for each input. With 28 options, every token shares a coarse expert pair with many other tokens. With 74 million options, tokens can be routed to genuinely different blends.
The combinatorial growth is what DeepSeek's ablations exploit. At the same active compute, finer segmentation consistently improves benchmark quality, because the model can decompose knowledge into smaller pieces and recompose them per token in ways the few large experts design cannot.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepSeek-V3 uses 256 routed experts plus shared experts with top-8 routing, the canonical fine-grained design.
- Qwen3-MoE 235B-A22B follows fine-grained segmentation with high expert count and high top-k.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf you double the expert count and halve expert size, what should you do to top-k?
Double top-k to keep active params constant. The combinatorial space grows roughly multiplicatively as a result.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking fine-grained experts cost more compute. At the same active parameter budget, per-token FLOPs are identical; the extra cost is routing decisions and dispatch bookkeeping, not arithmetic.
60 second bullets to scan on the way to the call.
How fine-grained segmentation keeps active parameters constant while slicing experts
Why combinatorial expressiveness is the quality lever, not raw expert count
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.