Without looking up the spec, predict how many transformer blocks (layers) a Llama style 7B and 70B model have, respectively. Report as 'NN / NN'.
Llama-style 7B models stack 32 transformer blocks; 70B models stack 80. Width and depth both scale, but width (`d_model`) scales faster.
Imagine building two skyscrapers, a small one and a big one. The small one (Llama 7B) is 32 floors tall and each floor is 4096 square meters wide. The big one (Llama 70B) is 80 floors tall and each floor is 8192 square meters wide: the floors got twice as wide AND there are more than twice as many of them. Engineers worked out a rule of thumb over years of experiments: when you make a language model bigger, you want to grow both the height (more layers) and the width of each layer, but width usually grows a bit faster. Llama 3 and Llama 3.1 follow this rule almost exactly. Memorizing 32 and 80 lets you sanity check almost any LLM cost estimate.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Two numbers (32 and 80) anchor most LLM-architecture conversations. They are the layer counts for the 7B-class and 70B-class Llama models, and they have stayed essentially fixed across Llama 1, 2, and 3.1. Memorizing them is not architecture trivia; it is the first input to almost every back of envelope cost or capacity calculation an LLM engineer does.
This question tests whether you can recall the canonical shape and explain why it landed there rather than at 24/96 or 40/80 or any other plausible pair. The answer connects to scaling-law work, to KV cache economics at serving time, and to the broader 'wider not deeper' trend that has defined frontier LLM design since Chinchilla.
The two anchor shapes
Llama 7B / 8B class: 32 layers, d_model = 4096. This shape was established by Llama 1 7B in February 2023, kept identically by Llama 2 7B, and inherited by Llama 3.1 8B. The 8B vs 7B difference is purely an embedding table cost from the vocab growth to 128k; the transformer stack is unchanged.
Llama 70B class: 80 layers, d_model = 8192. Same shape across all three Llama generations. The 70B-class FFN hidden dim is 28672 (about 3.5x d_model), and the model uses GQA with 8 KV heads.
The 7B and 70B shapes were not picked arbitrarily. They emerged from internal scaling-law sweeps that Meta has not fully published but whose conclusions track closely with the broader literature. At the 7B parameter budget, 32 layers / 4096 width sits very close to the compute optimal shape. At the 70B budget, 80 layers / 8192 width is similarly close to optimal.
The Mistral 7B model (2023) independently picked 32 / 4096 (the same shape) which is the strongest evidence that this is the right answer for 7B-class models, not just a Meta-internal preference.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3.1 8B: 32 layers, d_model 4096, 32 query heads with 8 KV heads (GQA 4:1), FFN hidden 14336.
- Llama 3.1 70B: 80 layers, d_model 8192, 64 query heads with 8 KV heads (GQA 8:1), FFN hidden 28672.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy did Llama 3.1 promote the 7B class to 8B without changing layer count or width?
The vocab grew from 32k to 128k, which alone adds about 800M parameters to the embedding tables. Layers and width were left alone because the scaling-law shape was already correct for that class.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Guessing 24 and 96 by analogy with GPT-3's small/large scaling, or assuming depth doubles cleanly with parameter count. Real LLMs scale width faster than depth.
60 second bullets to scan on the way to the call.
The canonical 7B layer count (32) and width (4096)
The canonical 70B layer count (80) and width (8192)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.