Vaswani 2017 set the FFN hidden dimension to 4x d_model and that ratio held for years. Modern SwiGLU-based models often use ~2.67x instead. Defend the original 4x choice, then explain what specifically changed with SwiGLU.
Free · 2 AI evals / day
**The 4x rule.** In the 2017 Transformer paper, the FFN's hidden dimension was set to `d_ff = 4 * d_model`. So if d_model is 512, the FFN expands the token through a 2048-dim hidden layer before projecting back to 512. That 4x ratio came out of the original ablations as a sweet spot: less than 4x left useful capacity on the table, more than 4x stopped improving quality at fixed compute.
It also has a clean parameter-counting consequence. The FFN holds two weight matrices, each of size `d_model * (4 * d_model) = 4 * d_model^2`. Total FFN params per block: `8 * d_model^2`. Attention holds four projections each of size `d_model^2`, total `4 * d_model^2`. So the FFN is twice the attention's parameter cost, which means roughly two-thirds of the block parameters live in the FFN. That ratio is one of the most-cited rules of thumb in transformer architecture.
**What SwiGLU changed.** A ReLU FFN has two weight matrices: an up-projection from d_model to d_ff, and a down-projection back. SwiGLU adds a third matrix called the gate. The forward pass becomes:
```
hidden = SwiGLU(x) = (W_up . x) * SiLU(W_gate . x)
out = W_down . hidden
```
The gate is multiplied elementwise with the up-projection before the down-projection. That third matrix is the same size as the up-projection, so SwiGLU's parameter count at a given hidden dim is 50% higher than ReLU's.
**Why the ratio dropped to ~2.67.** Authors want block parameter counts to stay comparable across model generations (so scaling-law comparisons remain valid). To match a 4x ReLU FFN's parameters, a SwiGLU FFN at the same d_ff would cost 1.5x as many. So they shrink d_ff. The arithmetic: `4 * 2 = 8` (ReLU at 4x has 8 d_model^2 parameters), and `2.67 * 3 = 8.01` (SwiGLU at 2.67x has three matrices each of size `d_model * (2.67 * d_model)`, total ~8 d_model^2 parameters). The expansion ratio looks smaller, but the parameter budget is the same.
**Defending the 4x rule.** Vaswani 2017's FFN used `d_ff = 4 * d_model`. The number was picked by ablation, not by theory; it landed as the sweet spot where adding more hidden capacity stopped improving validation loss at fixed compute. It survived as a near-universal default for five years (BERT, GPT-1/2/3, T5, original Llama) because the ablation kept reproducing across scales.
The 4x choice also produced one of the most-cited heuristics in transformer architecture: the FFN holds roughly **two-thirds of block parameters**. The arithmetic is:
- Attention projections (Q, K, V, O): `4 * d_model^2` parameters total.
- FFN at 4x: two matrices of shape `d_model * (4 * d_model)`, total `8 * d_model^2`.
- Block total: `12 * d_model^2`, of which 8 are FFN, 4 are attention.
So about 2/3 of dense-block parameters live in the FFN. Anyone optimizing inference cost or planning a parameter budget at scale starts from this ratio.
**What SwiGLU changed structurally.** SwiGLU is a Gated Linear Unit variant: instead of one up-projection followed by an activation, it uses two parallel projections (up and gate), applies SiLU (= swish, x * sigmoid(x)) to the gate, multiplies elementwise with the up-projection, and then down-projects:
```
hidden = (W_up . x) * SiLU(W_gate . x)
out = W_down . hidden
```
That is three weight matrices instead of two. At a given hidden dim, SwiGLU costs 50% more parameters than ReLU.
**Why ~2.67x is the new ratio.** Authors want apples to apples parameter comparison across architectures. To hold block parameter count constant when adopting SwiGLU, the hidden ratio must shrink. The math is direct:
- ReLU at 4x: 2 matrices * 4 * d_model^2 = 8 * d_model^2 params.
- SwiGLU at r * d_model: 3 matrices * r * d_model^2 = 3r * d_model^2 params.
- Set equal: 3r = 8, r = 8/3 ~= 2.67.
Llama 1/2/3/4 all use ~2.67x. Mistral uses 2.67x. Qwen 2/3 uses similar.
Quick reference
TL;DR
4x is the historical FFN expansion ratio from Vaswani 2017, picked empirically as the sweet spot for capacity vs compute and the source of the two thirds of block params rule.
Imagine the FFN is a small room where each word goes in, gets handed off across the room, and comes back out. The 2017 design had two big doors: one expanding the word into a wider workspace, one shrinking it back. The 'wide' part was 4 times wider than the input, because that was the size where adding more space stopped helping. SwiGLU added a third door: a gate that decides which parts of the wide workspace actually count. Three doors cost more lumber than two doors at the same width, so the designers made the room narrower (2.67x instead of 4x) to keep the total wood the same. The new room has three doors and is narrower; the old room had two doors and was wider. Both rooms cost about the same to build.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tiny models under 100M parameters where the FFN is already small The complexity overhead of SwiGLU's third matrix and elementwise multiply may not pay off at small scale; ReLU or GELU FFNs are simpler and adequate.
Inference workloads where elementwise multiply throughput is a bottleneck Some highly-fused inference kernels are tuned for ReLU or GELU activations. SwiGLU adds an elementwise op that may not fuse as cleanly on certain accelerators.
Research comparisons that need to match a specific baseline architecture exactly If the baseline is BERT or original-Llama, swapping to SwiGLU changes the parameter count and breaks the comparison; matching the original FFN shape is the safer call.
MoE blocks where the FFN is replaced by expert routing In MoE, each expert is a smaller FFN; the gating mechanism of SwiGLU may overlap awkwardly with the expert router. Some designs use plain GELU experts to keep the gating semantics clean.
Quick scan · 2–4 min
Everything important, quickly.
5 min: where the 4x rule came from, the two-thirds parameter heuristic, SwiGLU's three-matrix structure, the 8/3 parameter-parity arithmetic, why some models pick 3x or 3.5x instead, the empirical quality win.
Side-by-side
Property
ReLU FFN (Vaswani 2017)
SwiGLU FFN (modern)
Weight matrices per FFN
2 (up, down)
3 (gate, up, down)
Hidden ratio (d_ff / d_model)
4x
~2.67x (or 3-3.5x)
Parameter cost per FFN
8 * d_model^2
~8 * d_model^2 (parity by design)
FLOPs per FFN forward pass
2 matmuls + 1 activation
3 matmuls + 1 activation + 1 elementwise multiply
Block parameter share
~2/3 in FFN
~2/3 in FFN (preserved)
Quality vs ReLU baseline
Baseline
~1-2% perplexity reduction
Real-world examples
Real products, models, and research that use this idea.
5Qwen 2.5 / Qwen 3 (2024-2025): SwiGLU with ratios near 2.5-3x depending on size class.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is 8/3 ~= 2.67 the canonical starting point but several frontier models use 3x or 3.5x instead?
Sign in or go Pro to see the approach for this follow-up.
QCould you achieve the same quality gain as SwiGLU by just widening a ReLU FFN to use 50% more parameters?
Sign in or go Pro to see the approach for this follow-up.
QWhy did GELU not survive even though it was a smoother improvement on ReLU?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Citing 'SwiGLU is just better' as the explanation for the smaller hidden ratio. The ratio dropped specifically to keep parameter counts comparable; the quality improvement is a separate effect that emerges from the gating mechanism itself.