Predict the typical layer count for a Llama 7B and a Llama 70B

Llama 7B uses 32 transformer blocks. Llama 70B uses 80. Those two numbers have been the canonical reference points across Llama 1, 2, and 3.1 (the 7B-class is now Llama 3.1 8B, which keeps the same 32 layers). When the model gets 10x bigger in parameter count, **the architecture grows on two axes**: depth (number of layers) and width (`d_model`, the hidden dimension). Width grew from 4096 to 8192, a 2x increase. Depth grew from 32 to 80, a 2.5x increase. Both axes scale, but width tends to scale faster relative to early-2017 transformers because scaling-law work has consistently shown wider models pay off better at large parameter budgets. These specific numbers are worth memorizing. Almost any back of envelope estimate (KV cache size, FLOPs per token, parameter count) needs them as inputs. When someone asks 'how big is a 70B model?', the answer 'd_model 8192, 80 layers' is more useful than '70 billion parameters'.

**Llama 7B: 32 layers, `d_model = 4096`. Llama 70B: 80 layers, `d_model = 8192`.** These are the canonical reference shapes across Llama 1, 2, and 3.1 (Llama 3.1 promoted the 7B-class to 8B with the same 32-layer / 4096-width shape). The ratio matters more than the absolute numbers. Width 2x, depth 2.5x for a 10x parameter jump means width is growing slightly faster per doubling. That tracks with what scaling-law work has been telling us since Levine et al. and Chinchilla: at large parameter budgets, the compute optimal shape tends wider than the depth first intuition from 2017-era transformers. The parameter math sanity checks. With `d_model = 4096`, 32 layers, FFN hidden 14336, and GQA with 8 KV heads, you land at ~8B params. With `d_model = 8192`, 80 layers, FFN hidden 28672, and the same GQA ratio, you land at ~70B. The numbers are not arbitrary; they emerge from picking the compute optimal shape at each scale. Memorize 32 / 80. They are the two anchor points anyone working on LLM serving cost, KV cache sizing, or parameter estimation reaches for first.