A null expert slot is a deliberate no-op: the token skips FFN matmul and passes through unchanged via the residual connection.
A null expert is like a 'skip' button on the FFN layer. The router can send a token there when no specialist transformation is needed. The token keeps its current hidden state and flows through the residual path: no matmul, no zero vector injected.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Null (skip) expert slots are a Switch Transformer design pattern that lets the MoE router allocate zero FFN compute to certain tokens. The MCQ tests whether you understand what physically happens in the transformer block when a token lands on a null slot, a question that trips up candidates who conflate null experts with token dropping or fallback routing.
The correct answer is architectural, not heuristic: the FFN sublayer contributes nothing, and the residual connection passes the token through. Each wrong option describes a different MoE failure mode or engineering workaround.
This deep dive walks through the mechanism, disambiguates the distractors, and places null experts in the Switch/GShard lineage.
Null experts confuse three adjacent ideas: skip routing, overflow drops, and serving failover. The MCQ tests the architectural skip, FFN matmul does not run, residual carries identity. Switch Transformer lineage makes this still interview-relevant even if Mixtral and DeepSeek omit null slots.
MCQ discipline: map each option to a distinct failure mode, skip, drop, failover, zero-inject, only skip matches null architecture.
What a null expert slot is
In Switch-style MoE, the expert bank includes both real expert FFNs and one or more null (skip) slots. The router treats null slots as valid routing targets alongside real experts. When a token is routed to a null slot, the expert FFN matmul is not executed.
The MoE layer output is normally a weighted combination of expert FFN outputs. A null expert's output is identically zero, it has no weight matrices to apply. The token's hidden state before the FFN sublayer is preserved through the residual connection.
This gives the router a learned choice: transform this token through a specialist FFN, or skip FFN compute entirely. It is a form of conditional computation within the MoE framework.
Null slot competes in the same softmax as real experts, router learns when FFN transform is unnecessary. Padding and punctuation tokens are classic beneficiaries; saves FFN FLOPs without removing attention compute.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Switch Transformer (Fedus et al.) includes skip/null expert slots in the expert bank.
- GShard heritage capacity management uses different mechanisms (overflow dropping) than null experts.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy would a router learn to send tokens to null experts?
Some tokens may need only attention mixing, not FFN transformation, null slots save FLOPs on those tokens.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing null experts with capacity-overflow token dropping or fallback re-routing.
60 second bullets to scan on the way to the call.
Null expert = no-op FFN slot
Residual connection carries token through
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.