Which sublayer of a dense transformer block carries more parameters, attention or FFN?

**The FFN holds about two-thirds of block parameters.** This is one of the most useful facts about transformer architecture and one of the most surprising for people who learned about attention first. **The math.** Take a standard block at hidden size d_model with a 4x FFN expansion. - Attention has four projection matrices: Q, K, V, O. Each is `d_model x d_model`. Total: `4 * d_model^2`. - FFN has two projection matrices: up (`d_model -> 4 * d_model`) and down (`4 * d_model -> d_model`). Total: `2 * d_model * (4 * d_model) = 8 * d_model^2`. - Block total: `12 * d_model^2`. - FFN share: `8 / 12 = 2/3`. So attention contributes 4 d_model^2 parameters, the FFN contributes 8 d_model^2 parameters, and the FFN is twice as large. **Why people get this wrong.** Attention is the headline feature of the transformer. Every diagram, every blog post, and every introductory tutorial spends most of its time on attention. The FFN is usually described as 'a couple of linear layers' and gets a single sentence. That coverage imbalance creates the intuition that attention must be the bigger sublayer in terms of parameters, when actually it is the smaller one. **Why this matters in practice.** When you are optimizing inference cost, the FFN is where most of the weight memory lives, so quantization and MoE techniques target it first. When you are estimating model size from a config file, the FFN is the dominant term in your back of the envelope.

**The FFN carries roughly two-thirds of dense-block parameters.** Attention carries the other third. **The accounting.** At hidden size d_model with a 4x FFN expansion (Vaswani 2017 default): - Attention projections (Q, K, V, O): each `d_model x d_model`, so `4 * d_model^2` total. - FFN matrices (up, down): each `d_model * (4 * d_model)`, so `8 * d_model^2` total. - Per-block parameter total: `12 * d_model^2`. - FFN's share: 8/12 = 2/3. Attention's share: 4/12 = 1/3. This ratio is one of the most-cited rules of thumb in transformer architecture. It is exact for the 2017 design and approximately right for modern designs that swap ReLU for SwiGLU (which adds a third FFN matrix but compensates by shrinking the hidden dim to ~2.67 * d_model, preserving the same total parameter count). **Why the other options miss.** - *Option A inverts the ratio.* The four d_model x d_model attention projections are the smaller term, not the larger one. The intuition that attention is the 'big' sublayer comes from how much paper space it gets, not from its parameter count. - *Option C claims parity.* Parity only happens at FFN expansion 2x, which no production transformer uses. At 4x or higher, the FFN dominates. - *Option D confuses parameters with compute.* Block parameter counts are static, set by `d_model` and the FFN ratio. Sequence length affects FLOPs (because attention is O(n^2)) and KV cache size (which grows with n), but it does not change the parameter count of the weight matrices themselves. **Consequence for production.** Inference cost optimization (quantization, expert routing in MoE, distillation) targets the FFN first because that is where the weight memory lives. KV cache optimization (GQA, MQA, MLA, paged attention) targets attention because that is where the per token memory bandwidth lives. The two optimization targets are different physical phenomena and they correspond to different parts of the block.