Compute the trainable parameter count for a Llama style 7B decoder
A Llama style decoder has: d_model = 4096, n_layers = 32, n_heads = 32, head_dim = 128, FFN hidden = 11008 (SwiGLU, three matrices), vocab_size = 32000, untied embeddings, RMSNorm (1 scale vector per norm). KV heads = n_heads (full MHA, not GQA). Compute the total parameter count, rounded to the nearest 0.1B.
About 6.7B. Per block: attention ~67M + FFN ~135M ≈ 202.4M; 32 layers gives ~6.48B; plus untied embeddings (~262M) and norms ≈ 6.74B total. This is the canonical 'Llama-7B is actually 6.7B' figure.
Think of the model as 32 floors of a building plus an entry hall and an exit hall. Each floor has two rooms (attention at ~67M params and FFN at ~135M params) so a floor is about 202M. Stack 32 floors and you get about 6.48B. The entry hall is the input phone book that turns words into number lists (~131M). The exit hall is the output head that turns those number lists back into per-word scores (another ~131M because the head is its own separate table). Add a few small fixtures (norms) and the total lands at ~6.7B. That is why every 'Llama-7B' file you download is actually 6.7B parameters. The 7B is rounding to one significant figure.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Predicting the parameter count of a transformer is straightforward once you know the block structure. The trap is forgetting which formula applies (SwiGLU vs ReLU, MHA vs GQA, tied vs untied embeddings).
The standard 'Llama-7B' figure is actually 6.7B. The marketing number rounds up. The model file you download has 6,738,415,616 parameters exactly.
The per-block decomposition
A standard decoder block has two sublayers: attention and FFN. Each has its own parameter contribution. Norms are negligible.
Attention. It has four projection matrices: Q, K, V, and the output projection O. In standard MHA where the total query head width equals the model width (n_heads times head_dim is 4096), each projection is square. The attention parameter count per block is:
Llama is bias-free, so no bias parameters are added. Under GQA with fewer KV heads than query heads, K and V shrink. This model uses full MHA, so the square form applies.
FFN with SwiGLU. SwiGLU uses three matrices instead of two:
The three matrices are a gate projection up to FFN width, a value projection up to FFN width, and a projection back down to the model width. The SwiGLU activation multiplies the SiLU-activated gate stream by the value stream, then projects the result down. The 3x cost compared to a ReLU FFN's 2x is the price of the gated activation.
Why d_ffn = 11008. A standard ReLU FFN with the 4x expansion ratio has about 2 times 4096 times 16384, which is roughly 134.2M params. To keep the SwiGLU FFN at the same parameter budget, you want 3 times 4096 times d_ffn to also land near 134M, which gives d_ffn ≈ 11008. The seemingly arbitrary 11008 is a parameter preservation constant, not a mystical hyperparameter.
Block total. Per block params ≈ 67M + 135M ≈ 202.4M. Add 8K for two RMSNorms (one scale vector at model width each). The norms are below detection in the headline number.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
def llama_param_count(
d_model: int,
n_layers: int,
d_ffn: int,
vocab: int,
n_heads: int = None,
n_kv_heads: int = None,
head_dim: int = None,
tied: bool = False,
) -> int:
head_dim = head_dim or (d_model // n_heads)
n_kv_heads = n_kv_heads or n_heads
q_proj = d_model * (n_heads * head_dim)
kv_proj = 2 * d_model * (n_kv_heads * head_dim)
o_proj = (n_heads * head_dim) * d_model
attn = q_proj + kv_proj + o_proj
ffn = 3 * d_model * d_ffn # SwiGLU
norms = 2 * d_model # 2 RMSNorm per block
block = attn + ffn + norms
embed = vocab * d_model
head = 0 if tied else vocab * d_model
final_norm = d_model
return n_layers * block + embed + head + final_norm
# Llama 7B (full MHA, untied)
print(llama_param_count(4096, 32, 11008, 32000,
n_heads=32, head_dim=128, tied=False))
# -> 6738415616 (~6.7B)| Component | Formula | Llama 7B value |
|---|---|---|
| Attention per layer | 4 · d² | ~67M |
| FFN per layer (SwiGLU) | 3 · d · d_ffn | ~135M |
| RMSNorms per layer | 2 · d | ~8K (negligible) |
| Layers × per-layer | n_layers · (above) | ~6.48B |
| Input embedding | vocab · d | ~131M |
| Output head (untied) | vocab · d | ~131M |
| Final RMSNorm | d | ~4K |
| Grand total | - | ~6.74B (= '7B') |
Real products, models, and research that use this idea.
- Llama 3.1 8B: d=4096, n_layers=32, d_ffn=14336 (SwiGLU), vocab=128256, GQA with 8 KV heads. Per-block ~67M attn (full MHA equivalent) but actually less under GQA, plus ~176M FFN. Total ~8.0B.
- Llama 3.1 70B: d=8192, n_layers=80, d_ffn=28672 (SwiGLU), vocab=128256, GQA-8. Per-block ~151M attn (under GQA) + ~704M FFN. Total ~70B.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the formula change under GQA?
K and V projections shrink to (d_model, n_kv_heads · head_dim). For Llama 3 70B (d_model = 8192) with n_kv_heads = 8, K+V drop from ~134M per layer to ~17M per layer. Q and O stay square.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Using a two-matrix ReLU-style FFN formula instead of SwiGLU's three matrices, or assuming embeddings are tied (cuts ~131M off the count and lands at ~6.6B).
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.