Predict the trainable parameter count for a Llama style 7B decoder block by block
A Llama style decoder has: d_model = 4096, n_layers = 32, n_heads = 32, head_dim = 128, FFN hidden = 11008 (SwiGLU, three matrices), vocab_size = 32000, untied embeddings, RMSNorm (1 scale vector per norm). KV heads = n_heads (full MHA, not GQA). Predict the total parameter count, rounded to the nearest 0.1B.
About 6.7B. Per block: attention ~67M + FFN ~135M ≈ 202M; 32 layers gives ~6.46B; plus untied embeddings (~262M) and norms ≈ 6.72B total. This is the canonical 'Llama-7B is actually 6.7B' figure.
Think of the model as 32 floors of a building plus an entry hall and an exit hall. Each floor has two rooms (attention at ~67M params and FFN at ~135M params) so a floor is about 202M. Stack 32 floors and you get 6.46B. The entry hall is the input phone book that turns words into number lists (~131M); the exit hall is the output head that turns those number lists back into per-word scores (another ~131M because the head is its own separate table). Add a few small fixtures (norms) and the total lands at ~6.7B. That is why every 'Llama-7B' file you download is actually 6.7B parameters; the 7B is rounding to one significant figure.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Predicting the parameter count of a transformer is one of the cleanest interview signals. It separates candidates who memorized 'Llama is 7B parameters' from those who know what those parameters are doing. The arithmetic is straightforward once you know the block structure; the trap is forgetting which formula applies (SwiGLU vs ReLU, MHA vs GQA, tied vs untied embeddings).
The canonical 'Llama-7B' figure is actually 6.7B. The marketing number rounds up; the model file you download has 6,738,431,488 parameters exactly. Getting from the architecture spec to that exact number is what this question tests.
The per-block decomposition
A standard decoder block has two sublayers: attention and FFN. Each has its own parameter contribution; norms are negligible.
Attention. Four projection matrices: Q, K, V, and the output projection O. In standard MHA where the total query head width equals the model width (n_heads times head_dim is 4096), each projection is square. The attention parameter count per block is:
Llama is bias-free, so no bias parameters are added. Under GQA with fewer KV heads than query heads, K and V shrink; but this question specifies full MHA, so we use the square form.
FFN with SwiGLU. SwiGLU uses three matrices instead of two:
The three matrices are a gate projection up to FFN width, a value projection up to FFN width, and a projection back down to the model width. The SwiGLU activation multiplies the SiLU-activated gate stream by the value stream, then projects the result down. The 3x cost compared to a ReLU FFN's 2x is the price of the gated activation.
Why d_ffn = 11008. A standard ReLU FFN with the 4x expansion ratio has about 2 times 4096 times 16384, which is roughly 134.2M params. To keep the SwiGLU FFN at the same parameter budget, you want 3 times 4096 times d_ffn to also land near 134M, which gives d_ffn ≈ 11008. The seemingly arbitrary 11008 is a parameter-preservation constant, not a mystical hyperparameter.
Block total. Per-block params ≈ 67M + 135M ≈ 202M. Add 8K for two RMSNorms (one scale vector at model width each). The norms are below detection in the headline number.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
def llama_param_count(
d_model: int,
n_layers: int,
d_ffn: int,
vocab: int,
n_heads: int = None,
n_kv_heads: int = None,
head_dim: int = None,
tied: bool = False,
) -> int:
head_dim = head_dim or (d_model // n_heads)
n_kv_heads = n_kv_heads or n_heads
q_proj = d_model * (n_heads * head_dim)
kv_proj = 2 * d_model * (n_kv_heads * head_dim)
o_proj = (n_heads * head_dim) * d_model
attn = q_proj + kv_proj + o_proj
ffn = 3 * d_model * d_ffn # SwiGLU
norms = 2 * d_model # 2 RMSNorm per block
block = attn + ffn + norms
embed = vocab * d_model
head = 0 if tied else vocab * d_model
final_norm = d_model
return n_layers * block + embed + head + final_norm
# Llama 7B (full MHA, untied)
print(llama_param_count(4096, 32, 11008, 32000,
n_heads=32, head_dim=128, tied=False))
# -> 6738431488 (~6.7B)| Component | Formula | Llama 7B value |
|---|---|---|
| Attention per layer | 4 · d² | ~67M |
| FFN per layer (SwiGLU) | 3 · d · d_ffn | ~135M |
| RMSNorms per layer | 2 · d | ~8K (negligible) |
| Layers × per-layer | n_layers · (above) | ~6.46B |
| Input embedding | vocab · d | ~131M |
| Output head (untied) | vocab · d | ~131M |
| Final RMSNorm | d | ~4K |
| Grand total | - | ~6.72B (= '7B') |
Real products, models, and research that use this idea.
- Llama 3.1 8B: d=4096, n_layers=32, d_ffn=14336 (SwiGLU), vocab=128256, GQA with 8 KV heads. Per-block ~67M attn (full MHA equivalent) but actually less under GQA, plus ~176M FFN. Total ~8.0B.
- Llama 3.1 70B: d=8192, n_layers=80, d_ffn=28672 (SwiGLU), vocab=128256, GQA-8. Per-block ~67M attn (under GQA) + ~704M FFN. Total ~70B.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the formula change under GQA?
K and V projections shrink to (d_model, n_kv_heads · head_dim). For Llama 3 70B with n_kv_heads = 8, K+V drop from ~33M per layer to ~8M per layer. Q and O stay square.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Using a two-matrix ReLU-style FFN formula instead of SwiGLU's three matrices, or assuming embeddings are tied (cuts ~131M off the count and lands at ~6.6B).
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.