A colleague sketches a GPT-2-medium style decoder: 24 layers, d_model 1024, standard MHA (4d^2 attention params per block) and a 4x GELU FFN (8d^2 per block). Using the classic non-embedding estimate of 12 x L x d_model^2, calculate the approximate non-embedding parameter count.
Per transformer block, attention costs 4d^2 and the 4x FFN costs 8d^2, so 12 x 24 x 1024^2 lands at about 302M non-embedding parameters.
Think of each transformer layer as a small factory built from two workshops. The attention workshop runs four square machines, each costing d-by-d planks of wood. The feed-forward workshop briefly opens up to four times the width, then comes back down, costing the equivalent of eight d-by-d boards. So one factory uses twelve d-by-d boards in total. You have twenty four such factories stacked on top of each other. Each board is a thousand by a thousand cells (around a million cells). You need twelve of them per floor across twenty four floors. That gives about three hundred million cells of stored knowledge inside the stack. And that is before you count the dictionary that turns words into vectors.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Almost every transformer scaling discussion starts with the same back of the envelope number: roughly twelve times the number of layers times the model dimension squared. It is the formula a researcher will scribble on a whiteboard before they touch a config file. You can derive it in under a minute.
Where the 12 comes from: counting the matrices per block
Consider one decoder block in a standard pre-norm transformer. The block holds two sublayers: multi-head attention and a feed-forward network. Both read from and write to a residual stream of width d_model, written d for short.
Attention has four learnable matrices. The query, key, and value projections each map from d to d. The output projection, applied after the per head softmax and weighted sum operation, also maps from d to d. Four matrices of size d x d give 4d^2 parameters. The split into heads is only a reshape of these matrices and does not change the parameter count.
The feed-forward network is a two-layer MLP. It expands from d to 4d, applies a non-linearity (GELU in the original GPT-2 design, SwiGLU in modern stacks), then contracts back from 4d to d. The two matrices are sized d x 4d and 4d x d, each contributing 4d^2 parameters, for a total of 8d^2.
Sum the two sublayers: 4d^2 from attention, 8d^2 from the FFN, 12d^2 per block.
Multiplying by the number of blocks L gives the headline formula:
Situations where this technique stops working.
2–4 min · Everything important, quickly.
def non_emb_params(L: int, d_model: int) -> int:
attn = 4 * d_model ** 2 # Q, K, V, O projections
ffn = 2 * d_model * (4 * d_model) # up + down, 4x expansion
return L * (attn + ffn)
# GPT-2 medium style: 24 layers, d_model 1024
print(non_emb_params(24, 1024)) # 301_989_888 → about 302MReal products, models, and research that use this idea.
- GPT-2 medium uses L=24, d=1024 and lands at about 302M non-embedding, 355M total after the 50k vocab embedding
- Llama 3 8B has L=32, d=4096; 12 x 32 x 4096^2 is about 6.4B for the dense layers, with GQA shaving a sliver and the untied 128k vocab embedding and output head adding about 1.05B
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does GQA with 8 KV heads on a 32-head model change the per-block parameter count at d_model 4096?
K and V projections shrink from d x d to d x d_kv where d_kv = (8/32) d. Compute the new attention total (Q at 4096x4096, O at 4096x4096, K and V at 4096x1024 each) and subtract from 4d^2.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forgetting that the FFN is twice the attention cost (8d^2 vs 4d^2), or quietly adding the embedding table into a count the rule explicitly excludes.
60 second bullets to scan on the way to the call.
Why attention contributes exactly 4d^2 per block
Why a 4x FFN contributes 8d^2 and not 4d^2
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.