Spend a fixed parameter budget: more layers or wider d_model, and what does the evidence say?

At a **fixed parameter budget**, you should generally spend the marginal parameter on width (`d_model`) rather than depth (more layers) past a certain threshold. **The empirical anchors** from scaling law work: - 7B class models: about 32 layers is the depth sweet spot. Adding more layers past that gives diminishing quality returns. - 70B class models: about 80 layers. Llama 70B lands here. - 405B class: about 126 layers (Llama 3.1 405B). Depth grew sub-linearly with parameter count. **The intuition**: depth provides representational composition (each layer is a refinement step that builds on the previous one). Width provides per step capacity: how much information each layer can hold and process. Once you have enough refinement steps, additional steps don't help; the bottleneck shifts to how much each step can do. **Modern Llama 70B** uses `d_model = 8192` with 80 layers. That is wider than the 2017 transformer ratios would suggest for the same parameter count, reflecting the wider not deeper finding from years of scaling law sweeps.

**At a fixed parameter budget, scale width past the depth sweet spot.** The depth threshold scales sub-linearly with parameter count: ~32 layers at 7B, ~80 at 70B, ~126 at 405B. **The empirical evidence.** Levine et al. (2020, 'Limits to Depth Efficiencies') showed analytically and empirically that past a certain depth, additional layers contribute less than the equivalent width increase. Chinchilla era follow-ups confirmed this when training compute became the primary constraint. The Kaplan et al. and Hoffmann et al. scaling law papers calibrated the relationship between parameters, data, and compute, and recent industrial sweeps (Meta's Llama 3 paper, in particular) reported aspect ratios that match the wider not deeper prescription closely. **The intuition: composition vs capacity.** Depth provides representational composition: each layer refines features built by previous layers. Width provides per step capacity, how much information each layer can encode. At small parameter budgets, composition is the bottleneck; adding layers gives the model more refinement steps and quality scales cleanly. At large parameter budgets, capacity per step becomes the bottleneck, and the marginal parameter buys more quality if spent on `d_model`. **Modern frontier shapes.** Llama 3.1 70B: 80 layers, `d_model = 8192`, aspect ratio `d_model / n_layers ≈ 102`. Llama 3.1 405B: 126 layers, `d_model = 16384`, aspect ratio ≈ 130. The aspect ratio settles in the 100-130 range at frontier scale, well above the 2017 Vaswani ratio of `512 / 6 ≈ 85`. **Caveats.** Depth still matters for tasks requiring long reasoning chains; recent test time compute work (o1 style) leans on depth of reasoning rather than depth of layers but the analogy is informative. Tiny models (sub-1B) sometimes benefit from being deeper than the scaling law fit predicts because composition matters more at limited capacity.