Mixture of Experts (MoE)
Also known as: MoE, Sparse MoE
Scale model capacity by routing each token to a small subset of expert MLPs instead of using all of them.
An architecture where only a subset of expert sub-networks are activated per token, selected by a learned routing function. Enables scaling model capacity without proportional compute cost.
In practice
Powers Mixtral, GPT-4-class models, and DeepSeek. Senior interviews dig into routing instability, load balancing, and inference-time activation cost.
Related topics
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Transformer
The attention-only neural architecture behind GPT, Claude, Gemini, and almost every modern LLM.
Context Window
The max number of tokens a model can attend to at once.
RoPE (Rotary Position Embedding)
Position info injected by rotating Q and K vectors, easy to extend to longer contexts.
Multi-Head Attention (MHA)
Run several attention heads in parallel with different projections, then concat. This captures multiple relationship types per layer.
Layer Normalization
Per-sample, per-position normalization. The glue that keeps transformer training stable.