A null expert slot is a deliberate no-op: the token skips FFN matmul and passes through unchanged via the residual connection.
A null expert is like a 'skip' button on the FFN layer. The router can send a token there when no specialist transformation is needed. The token keeps its current hidden state and flows through the residual path: no matmul runs.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Null (skip) expert slots are a design pattern from later MoE research, such as AdaMoE (Zeng et al., 2024). They let the MoE router allocate zero FFN compute to certain tokens.
The mechanism is architectural, not heuristic: the FFN sublayer contributes nothing, and the residual connection passes the token through. Each wrong option describes a different MoE failure mode or engineering workaround.
Three adjacent ideas are easy to confuse: skip routing (null experts), overflow drops, and serving failover. Null experts are the architectural skip: the FFN matmul does not run, and the residual carries the identity. Mixtral and DeepSeek omit null slots.
Map each option to a distinct mechanism: skip, drop, failover, dense compute. Only skip describes the null architecture.
What a null expert slot is
In null-expert designs such as AdaMoE, the expert bank includes both real expert FFNs and one or more null (skip) slots. The router treats null slots as valid routing targets alongside real experts. When the router sends a token to a null slot, the expert FFN matmul does not run.
The MoE layer output is normally a weighted combination of expert FFN outputs. A null expert's output is identically zero: it has no weight matrices to apply. The token's hidden state before the FFN sublayer is preserved through the residual connection.
This gives the router a learned choice: transform this token through a specialist FFN, or skip FFN compute entirely. It is a form of conditional computation within the MoE framework.
The null slot competes in the same softmax as real experts, so the router learns when an FFN transform is unnecessary. Padding and punctuation tokens are classic beneficiaries. This saves FFN FLOPs without removing attention compute.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- AdaMoE (Zeng et al., 2024) adds null expert slots to the expert bank and raises top-k.
- GShard heritage capacity management uses different mechanisms (overflow dropping) than null experts.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy would a router learn to send tokens to null experts?
Some tokens may need only attention mixing, not FFN transformation. Null slots save FLOPs on those tokens.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing null experts with capacity-overflow token dropping or fallback re-routing.
60 second bullets to scan on the way to the call.
Null expert = no-op FFN slot
Residual connection carries token through
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.