Pick the statements that accurately describe how an MoE sublayer replaces a dense FFN

Mixture-of-Experts is a surgical change to a transformer block. It replaces only the **FFN sublayer**, and leaves attention, residual connections, and normalization untouched. In a dense block, the FFN is a single MLP: two or three matrices that every token passes through. In an MoE block, that single MLP is replaced by `N` parallel expert FFNs (8, 64, 128; the number varies by model). A small linear layer called the **router** scores each expert for each token, and the top-k experts (usually k=1 or k=2) are actually run on that token. Because only `k` of `N` experts run per token, the FLOPs stay close to a dense FFN. But because all `N` expert weight sets exist, total parameters grow roughly `N`x. The router's softmaxed scores combine the chosen experts' outputs into one vector that replaces what the dense FFN would have produced. The block recipe is otherwise identical: same pre-norm, same residual add. MoE is a sublayer swap, not a block rewrite.

**MoE is a sublayer-local swap.** The attention sublayer, residual wrap, and normalization placement are all unchanged. Only the dense FFN gets replaced. The replacement has two pieces. First, `N` parallel expert FFNs, each with the same shape as the dense FFN it replaces. Second, a tiny router (a single `(d_model, N)` linear layer plus softmax) that scores experts per token and selects the top-k. **Sparse activation is the entire point.** With `k=1` (Llama 4 Maverick, 128 experts, top-1) or `k=2` (Mixtral, 8 experts, top-2), each token runs through only a small fraction of the total parameters. The active FLOP count stays close to a dense FFN of the same per-expert size, while the total parameter count scales with `N`. **Two recurring misreadings.** First, MoE does not touch attention or the block scaffolding; the residual stream, pre-norm, and attention layer are bit-identical to a dense block. Second, sparse routing means tokens see only `k` experts, not all of them. If every token went through every expert (dense routing), there would be no FLOP savings and no reason to bother. The router learns which expert each token should visit through a load-balancing auxiliary loss during training, so expert utilization stays roughly uniform.