Derive the SwiGLU hidden width that matches a 4x GELU FFN parameter budget
A standard GELU FFN at d_model=4096 uses hidden width 4 x 4096 = 16384 across two matrices (up and down). You swap in SwiGLU, which needs THREE matrices (gate, up, down), but the param budget must stay identical. Predict the approximate hidden width the SwiGLU FFN should use, and name the width Llama-2-7B actually shipped.
Solve 3dh = 8d^2 for h: hidden width drops from 4d to (8/3) d, about 10923 at d=4096; Llama-2-7B rounds to 11008.
A plain FFN has two matrices that touch a hidden layer of width 4d. SwiGLU has three matrices touching the same hidden layer, so if you keep the width at 4d, you have used 50% more parameters. To stay at the same param budget, shrink the hidden width. Three matrices times d times h must equal two matrices times d times 4d. Solve for h and you get (8/3) d. At d=4096 that is about 10923. Hardware likes numbers that are multiples of 128 or 256, so Llama-2-7B rounds up to 11008. The ratio 8/3 is sometimes called the 'two-thirds rule' for gated FFNs.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The (8/3) d hidden width rule for SwiGLU FFNs is one of those small numbers that shows up in every modern LLM config and that nobody explains, even though it is a one-line derivation. The number is not magical, it is not empirically tuned, and it is not a hyperparameter. It is the unique hidden width that makes a three-matrix SwiGLU FFN cost exactly the same number of parameters as a two-matrix GELU FFN at the same d_model. Once you know the derivation, you can read every Llama config file at a glance.
This walkthrough does three things. It derives the rule, plugs in the numbers for the canonical example (Llama-2-7B), and then surveys where modern LLMs follow the rule strictly versus deliberately violate it.
Mental model: the rule is a budget-matching constraint, not a quality optimum. Modern models often violate it on purpose to spend extra params on FFN capacity. The rule is the right baseline for ablations and the right starting point for design.
The derivation
Plain GELU FFN
Two matrices: up [d, h] and down [h, d]. Hidden width h = 4d by convention. Total parameters:
At d = 4096, P = 8 * 4096^2 = 134,217,728, about 134M.
SwiGLU FFN
Three matrices: gate [d, h], up [d, h], down [h, d]. Total parameters:
Set the two equal to keep the parameter budget fixed:
This is the two-thirds rule: SwiGLU's hidden width is two-thirds of the plain FFN's 4d, or equivalently (8/3) times d_model.
Plug in d = 4096
h = (8/3) * 4096 = 10,922.67
Llama-2-7B ships h = 11008, the next multiple of 256. Let us verify the params at the shipped width:
P = 3 * 4096 * 11008 = 135,266,304, about 135M, within 1% of the GELU FFN's 134M. The bump is rounding slack.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 2 7B at d=4096 ships h=11008, the canonical (8/3) d example rounded to a hardware-friendly number.
- Llama 2 13B at d=5120 ships h=13824, a multiple of 256 above (8/3) x 5120 = 13653.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the (8/3) rule change for MoE FFNs?
Per-expert width follows the same rule, but total FFN params scale with the number of experts. Activated params per token scale with top-k experts, not total experts, so effective compute uses a different multiplier than total params.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Keeping the SwiGLU hidden width at 4d and accepting a 50% param increase. The (8/3) d rebalance is what makes ablations between SwiGLU and GELU FFNs comparable at fixed param count.
60 second bullets to scan on the way to the call.
Derive the (8/3) rule from the equation 3 d h = 8 d^2
Compute h for d=4096 and round to a hardware multiple
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.