**The FFN holds about two-thirds of block parameters.** This is one of the most useful facts about transformer architecture and one of the most surprising for people who learned about attention first.
**The math.** Take a standard block at hidden size d_model with a 4x FFN expansion.
- Attention has four projection matrices: Q, K, V, O. Each is `d_model x d_model`. Total: `4 * d_model^2`.
- FFN has two projection matrices: up (`d_model -> 4 * d_model`) and down (`4 * d_model -> d_model`). Total: `2 * d_model * (4 * d_model) = 8 * d_model^2`.
- Block total: `12 * d_model^2`.
- FFN share: `8 / 12 = 2/3`.
So attention contributes 4 d_model^2 parameters, the FFN contributes 8 d_model^2 parameters, and the FFN is twice as large.
**Why people get this wrong.** Attention is the headline feature of the transformer. Every diagram, every blog post, and every introductory tutorial spends most of its time on attention. The FFN is usually described as 'a couple of linear layers' and gets a single sentence. That coverage imbalance creates the intuition that attention must be the bigger sublayer in terms of parameters, when actually it is the smaller one.
**Why this matters in practice.** When you are optimizing inference cost, the FFN is where most of the weight memory lives, so quantization and MoE techniques target it first. When you are estimating model size from a config file, the FFN is the dominant term in your back of the envelope.
**The FFN carries roughly two-thirds of dense-block parameters.** Attention carries the other third.
**The accounting.** At hidden size d_model with a 4x FFN expansion (Vaswani 2017 default):
- Attention projections (Q, K, V, O): each `d_model x d_model`, so `4 * d_model^2` total.
- FFN matrices (up, down): each `d_model * (4 * d_model)`, so `8 * d_model^2` total.
- Per-block parameter total: `12 * d_model^2`.
- FFN's share: 8/12 = 2/3. Attention's share: 4/12 = 1/3.
This ratio is one of the most-cited rules of thumb in transformer architecture. It is exact for the 2017 design and approximately right for modern designs that swap ReLU for SwiGLU (which adds a third FFN matrix but compensates by shrinking the hidden dim to ~2.67 * d_model, preserving the same total parameter count).
**Why the other options miss.**
- *Option A inverts the ratio.* The four d_model x d_model attention projections are the smaller term, not the larger one. The intuition that attention is the 'big' sublayer comes from how much paper space it gets, not from its parameter count.
- *Option C claims parity.* Parity only happens at FFN expansion 2x, which no production transformer uses. At 4x or higher, the FFN dominates.
- *Option D confuses parameters with compute.* Block parameter counts are static, set by `d_model` and the FFN ratio. Sequence length affects FLOPs (because attention is O(n^2)) and KV cache size (which grows with n), but it does not change the parameter count of the weight matrices themselves.
**Consequence for production.** Inference cost optimization (quantization, expert routing in MoE, distillation) targets the FFN first because that is where the weight memory lives. KV cache optimization (GQA, MQA, MLA, paged attention) targets attention because that is where the per token memory bandwidth lives. The two optimization targets are different physical phenomena and they correspond to different parts of the block.
Quick reference
TL;DR
In a standard dense transformer block, the FFN holds roughly two-thirds of the parameters and attention holds one-third.
Picture a transformer block as a workshop with two big machines. One machine, attention, takes every word's data and lets it talk to every other word. The other machine, the FFN, takes each word one at a time and reshapes it in a much wider workspace before squeezing it back to size. Now count the wires (the parameters). Attention has four boxes, each the same size as the input. The FFN has two boxes, but each one is four times wider than the input. So the FFN's two big boxes have more wires than attention's four small boxes. About two-thirds of the workshop's wiring lives in the FFN. The diagrams in papers always show attention as the headline feature, but the parameters live mostly in the FFN.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Quoting the 2/3 figure for very small models with small FFN ratios Some tiny models (under 50M params) shrink the FFN ratio to reduce footprint; the 2/3 rule does not hold.
Reasoning about per token compute at very long sequences Attention's O(n^2) compute starts to dominate FLOPs even though it is the minority of parameters. The parameter rule and the FLOPs rule disagree.
MoE total parameter accounting MoE shifts FFN share to near-total of model params; the 2/3 number is for dense blocks specifically.
Multimodal models with extra projection layers Image, audio, or video encoders attached to a transformer body shift the parameter accounting; the block-level 2/3 still holds but the model-level share changes.
Quick scan · 2–4 min
Everything important, quickly.
3 min: do the arithmetic, name the 2/3 rule, explain why it surprises people, mention how GQA and MoE shift the ratio further toward FFN dominance.
Side-by-side
Sublayer
Matrices
Parameter count (d_model = 4096, 4x FFN)
Share of block
Attention Q, K, V, O
4 matrices, each d_model x d_model
67.1M
~1/3
FFN up, down (ReLU)
2 matrices, d_model -> 4d_model and back
134.2M
~2/3
FFN gate, up, down (SwiGLU at 8/3 ratio)
3 matrices, d_model x 2.67*d_model
~134.2M
~2/3
Block total
(all of the above)
~201M
100%
Real-world examples
Real products, models, and research that use this idea.
1Llama 3 8B (d_model = 4096, d_ff = 14336): one block has ~67M attention params (GQA reduces this) and ~176M FFN params (SwiGLU, three matrices). FFN dominates.
2GPT-2 small (d_model = 768, d_ff = 3072): one block has 2.36M attention params and 4.72M FFN params. Exact 2:1 ratio.
3Mistral 7B (d_model = 4096, d_ff = 14336, GQA with 8 KV heads): FFN holds ~80% of block parameters once GQA is factored in.
4Mixtral 8x7B: dense attention plus 8 SwiGLU experts. Each block's FFN experts hold roughly 1.4B parameters together, dwarfing attention's ~30M.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the FFN's parameter dominance not translate into FFN dominance of inference latency at long context?
Sign in or go Pro to see the approach for this follow-up.
QHow does GQA change the parameter ratio between attention and FFN?
Sign in or go Pro to see the approach for this follow-up.
QIf FFN is the parameter-heavy sublayer, why did the 2017 paper not just make it the only sublayer?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Believing attention dominates the parameter count because it dominates the diagrams. Attention dominates the *FLOPs* at long context, but the *parameters* live mostly in the FFN.