Which sublayer of a dense transformer block carries more parameters, attention or FFN?

MCQ·Easy·4.0 · 0·~1 min·Asked atDatabricks Phonepe Sierra·Relevant atDeepseek Mistral AI

Attempt it

**The FFN holds about two-thirds of block parameters.** This is one of the most useful facts about transformer architecture and one of the most surprising for people who learned about attention first. **The math.** Take a standard block at hidden size d_model with a 4x FFN expansion. - Attention has four projection matrices: Q, K, V, O. Each is `d_model x d_model`. Total: `4 * d_model^2`. - FFN has two projection matrices: up (`d_model -> 4 * d_model`) and down (`4 * d_model -> d_model`). Total: `2 * d_model * (4 * d_model) = 8 * d_model^2`. - Block total: `12 * d_model^2`. - FFN share: `8 / 12 = 2/3`. So attention contributes 4 d_model^2 parameters, the FFN contributes 8 d_model^2 parameters, and the FFN is twice as large. **Why people get this wrong.** Attention is the headline feature of the transformer. Every diagram, every blog post, and every introductory tutorial spends most of its time on attention. The FFN is usually described as 'a couple of linear layers' and gets a single sentence. That coverage imbalance creates the intuition that attention must be the bigger sublayer in terms of parameters, when actually it is the smaller one. **Why this matters in practice.** When you are optimizing inference cost, the FFN is where most of the weight memory lives, so quantization and MoE techniques target it first. When you are estimating model size from a config file, the FFN is the dominant term in your back of the envelope.

**The FFN carries roughly two-thirds of dense-block parameters.** Attention carries the other third. **The accounting.** At hidden size d_model with a 4x FFN expansion (Vaswani 2017 default): - Attention projections (Q, K, V, O): each `d_model x d_model`, so `4 * d_model^2` total. - FFN matrices (up, down): each `d_model * (4 * d_model)`, so `8 * d_model^2` total. - Per-block parameter total: `12 * d_model^2`. - FFN's share: 8/12 = 2/3. Attention's share: 4/12 = 1/3. This ratio is one of the most-cited rules of thumb in transformer architecture. It is exact for the 2017 design and approximately right for modern designs that swap ReLU for SwiGLU (which adds a third FFN matrix but compensates by shrinking the hidden dim to ~2.67 * d_model, preserving the same total parameter count). **Why the other options miss.** - *Option A inverts the ratio.* The four d_model x d_model attention projections are the smaller term, not the larger one. The intuition that attention is the 'big' sublayer comes from how much paper space it gets, not from its parameter count. - *Option C claims parity.* Parity only happens at FFN expansion 2x, which no production transformer uses. At 4x or higher, the FFN dominates. - *Option D confuses parameters with compute.* Block parameter counts are static, set by `d_model` and the FFN ratio. Sequence length affects FLOPs (because attention is O(n^2)) and KV cache size (which grows with n), but it does not change the parameter count of the weight matrices themselves. **Consequence for production.** Inference cost optimization (quantization, expert routing in MoE, distillation) targets the FFN first because that is where the weight memory lives. KV cache optimization (GQA, MQA, MLA, paged attention) targets attention because that is where the per token memory bandwidth lives. The two optimization targets are different physical phenomena and they correspond to different parts of the block.

TL;DR

In a standard dense transformer block, the FFN holds roughly two-thirds of the parameters and attention holds one-third.

Memory aid

Easy to grasp

Picture a transformer block as a workshop with two big machines. One machine, attention, takes every word's data and lets it talk to every other word. The other machine, the FFN, takes each word one at a time and reshapes it in a much wider workspace before squeezing it back to size. Now count the wires (the parameters). Attention has four boxes, each the same size as the input. The FFN has two boxes, but each one is four times wider than the input. So the FFN's two big boxes have more wires than attention's four small boxes. About two-thirds of the workshop's wiring lives in the FFN. The diagrams in papers always show attention as the headline feature, but the parameters live mostly in the FFN.

Key concepts

Feed-forward network Attention Parameter counting

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

This is a deceptively important architecture question. The 'FFN holds two-thirds of block parameters' rule is the foundation of every back of the envelope parameter estimate, every inference-cost analysis, and every modern efficiency technique (quantization, MoE). Anyone who has worked on transformer scaling knows the number immediately. Anyone who has only read about attention often gets it backward.

This card walks through the arithmetic, explains why the intuition often fails, shows how GQA and MoE shift the ratio further toward FFN dominance, and connects the parameter accounting to where production optimization effort actually lands.

The arithmetic in three lines

For a standard dense block at hidden size d_model with a 4x FFN expansion ratio:

code

Attention Q, K, V, O projections: 4 * d_model^2
FFN up, down matrices:            2 * d_model * (4 * d_model) = 8 * d_model^2
Block total:                       12 * d_model^2

FFN share: 8 / 12 = 2/3. Attention share: 4 / 12 = 1/3.

A worked example at d_model = 4096:

Attention: 4 * 4096^2 = 67.1M parameters.
FFN: 8 * 4096^2 = 134.2M parameters.
Block total: 201.3M parameters.

Multiply by ~80 blocks for a 70B model, add the embedding table, and you get the right order of magnitude for a Llama-class architecture.

Why the intuition is often backward

How GQA, MQA, MLA, and MoE shift the ratio further

What this means in production

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Sublayer	Matrices	Parameter count (d_model = 4096, 4x FFN)	Share of block
Attention Q, K, V, O	4 matrices, each d_model x d_model	67.1M	~1/3
FFN up, down (ReLU)	2 matrices, d_model -> 4d_model and back	134.2M	~2/3
FFN gate, up, down (SwiGLU at 8/3 ratio)	3 matrices, d_model x 2.67*d_model	~134.2M	~2/3
Block total	(all of the above)	~201M	100%

Real products, models, and research that use this idea.

Llama 3 8B (d_model = 4096, d_ff = 14336): one block has ~67M attention params (GQA reduces this) and ~176M FFN params (SwiGLU, three matrices). FFN dominates.
GPT-2 small (d_model = 768, d_ff = 3072): one block has 2.36M attention params and 4.72M FFN params. Exact 2:1 ratio.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the FFN's parameter dominance not translate into FFN dominance of inference latency at long context?

Latency depends on both parameter count and per-position compute. FFN is per-position O(d_model * d_ff), so its FLOPs scale linearly with sequence length. Attention is O(n^2 * d_model), so its FLOPs scale quadratically. At short context the FFN dominates compute (matching parameter count). At long context attention's quadratic term overtakes. KV cache memory bandwidth also becomes the bottleneck at decode time for long sequences.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Believing attention dominates the parameter count because it dominates the diagrams. Attention dominates the *FLOPs* at long context, but the *parameters* live mostly in the FFN.

60 second bullets to scan on the way to the call.

FFN holds 2/3 of dense block parameters
Attention holds 1/3 of dense block parameters

Primary sources. Browse if you want the original framing.