Predict the trainable parameter count for a Llama-style 7B decoder block by block
A Llama-style decoder has: d_model = 4096, n_layers = 32, n_heads = 32, head_dim = 128, FFN hidden = 11008 (SwiGLU, three matrices), vocab_size = 32000, untied embeddings, RMSNorm (1 scale vector per norm). KV heads = n_heads (full MHA, not GQA). Predict the total parameter count, rounded to the nearest 0.1B.
About 6.7B. Per block: attention ~67M + FFN ~135M ≈ 202M; 32 layers gives ~6.46B; plus untied embeddings (~262M) and norms ≈ 6.72B total. This is the canonical 'Llama-7B is actually 6.7B' figure.
Think of the model as 32 floors of a building plus an entry hall and an exit hall. Each floor has two rooms (attention at ~67M params and FFN at ~135M params) so a floor is about 202M. Stack 32 floors and you get 6.46B. The entry hall is the input phone book that turns words into number lists (~131M); the exit hall is the output head that turns those number lists back into per-word scores (another ~131M because the head is its own separate table). Add a few small fixtures (norms) and the total lands at ~6.7B. That is why every 'Llama-7B' file you download is actually 6.7B parameters; the 7B is rounding to one significant figure.
Detailed answer & concept explanation~6 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5 min: write the three formulas (attention, FFN, embeddings), plug in numbers, total to ~6.72B, mention SwiGLU's three matrices and the tied-vs-untied question.
def llama_param_count(
d_model: int,
n_layers: int,
d_ffn: int,
vocab: int,
n_heads: int = None,
n_kv_heads: int = None,
head_dim: int = None,
tied: bool = False,
) -> int:
head_dim = head_dim or (d_model // n_heads)
n_kv_heads = n_kv_heads or n_heads
q_proj = d_model * (n_heads * head_dim)
kv_proj = 2 * d_model * (n_kv_heads * head_dim)
o_proj = (n_heads * head_dim) * d_model
attn = q_proj + kv_proj + o_proj
ffn = 3 * d_model * d_ffn # SwiGLU
norms = 2 * d_model # 2 RMSNorm per block
block = attn + ffn + norms
embed = vocab * d_model
head = 0 if tied else vocab * d_model
final_norm = d_model
return n_layers * block + embed + head + final_norm
# Llama 7B (full MHA, untied)
print(llama_param_count(4096, 32, 11008, 32000,
n_heads=32, head_dim=128, tied=False))
# -> 6738431488 (~6.7B)| Component | Formula | Llama 7B value |
|---|---|---|
| Attention per layer | 4 · d² | ~67M |
| FFN per layer (SwiGLU) | 3 · d · d_ffn | ~135M |
| RMSNorms per layer | 2 · d | ~8K (negligible) |
| Layers × per-layer | n_layers · (above) | ~6.46B |
| Input embedding | vocab · d | ~131M |
| Output head (untied) | vocab · d | ~131M |
| Final RMSNorm | d | ~4K |
| Grand total | - | ~6.72B (= '7B') |
Real products, models, and research that use this idea.
- Llama 3.1 8B: d=4096, n_layers=32, d_ffn=14336 (SwiGLU), vocab=128256, GQA with 8 KV heads. Per-block ~67M attn (full MHA equivalent) but actually less under GQA, plus ~176M FFN. Total ~8.0B.
- Llama 3.1 70B: d=8192, n_layers=80, d_ffn=28672 (SwiGLU), vocab=128256, GQA-8. Per-block ~67M attn (under GQA) + ~704M FFN. Total ~70B.
- Llama 3.1 405B: d=16384, n_layers=126, d_ffn=53248 (SwiGLU), vocab=128256, GQA. Per-block ~268M attn + ~2.6B FFN. Total ~405B.
- Mistral 7B (v0.3): d=4096, n_layers=32, d_ffn=14336, vocab=32768, GQA-8 (kv_heads=8). Total ~7.25B; the larger vocab and FFN hidden vs Llama-7B explain the small bump.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the embedding ratio so high in small models but negligible in large ones?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Using a two-matrix ReLU-style FFN formula instead of SwiGLU's three matrices, or assuming embeddings are tied (cuts ~131M off the count and lands at ~6.6B).
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.