Find the wrong claim about what MoE replaces in a transformer block

Zenaique

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Quick reference

TL;DR

MoE sparsifies the FFN sublayer only; self-attention stays dense and mixes across all token positions.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Think of a transformer block as two steps in a meeting. First, every word looks at every other word to share context (attention, everyone talks to everyone). Second, each word gets processed individually through a small neural network (the FFN). MoE replaces that second step with a team of specialist FFNs and a router, but the first step, where words share information, stays the same.

Key concepts

Mixture of Experts Attention Transformer Architecture

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

6 min: block anatomy + FFN vs attention roles + production MoE stacks + sparse attention distinction.

Real-world examples

Real products, models, and research that use this idea.

Mixtral 8×7B keeps dense sliding-window + full attention while MoE-sparsifying FFN layers.
DeepSeek-V3 uses dense MLA attention with fine-grained MoE FFN experts.
Switch Transformer paper explicitly applies MoE to the feed-forward layer only.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QCould you apply MoE to attention instead of FFN?

Sign in or go Pro to see the approach for this follow-up.

QWhere do shared experts sit in the block?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Claiming MoE replaces self-attention instead of the feed-forward sublayer.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.