Mixture of Experts (MoE)
Also known as: MoE, Sparse MoE
Scale model capacity by routing each token to a small subset of expert MLPs instead of using all of them.
An architecture where only a subset of expert sub-networks are activated per token, selected by a learned routing function. Enables scaling model capacity without proportional compute cost.
In practice
Powers Mixtral, GPT-4-class models, and DeepSeek. Senior interviews dig into routing instability, load balancing, and inference-time activation cost.
Related topics
Questions that mention this term
- Which best describes shared…
- Adding shared experts to a routed only MoE stack, which effects apply?
- Match DeepSeek MoE design choices to their descriptions
- Explain why your MoE server's cost per token falls sharply as batch size grows.
- How should a team disclose MoE model size for honest latency and cost comparisons?
- What best explains higher tail latency for MoE serving versus a dense model with similar active compute?
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Causal Mask
Attention mask that hides future tokens so each position can only see itself and prior tokens.
Context Window
The max number of tokens a model can attend to at once.
Decoder-Only
Single autoregressive transformer stack: the shape of every modern frontier LLM.
Encoder-Decoder
Transformer with separate encoder + decoder stacks; strong for translation and structured seq2seq tasks.
Layer Normalization
Per-sample, per-position normalization. The glue that keeps transformer training stable.