Given a fixed parameter budget for a new LLM, defend the choice between scaling depth (more layers) and scaling width (larger d_model). Reference what scaling law ablations from Levine et al. and the Chinchilla era follow-ups found, and how modern 70B class models tend to land.
Free · 2 AI evals / day
At a **fixed parameter budget**, you should generally spend the marginal parameter on width (`d_model`) rather than depth (more layers) past a certain threshold.
**The empirical anchors** from scaling law work:
- 7B class models: about 32 layers is the depth sweet spot. Adding more layers past that gives diminishing quality returns.
- 70B class models: about 80 layers. Llama 70B lands here.
- 405B class: about 126 layers (Llama 3.1 405B). Depth grew sub-linearly with parameter count.
**The intuition**: depth provides representational composition (each layer is a refinement step that builds on the previous one). Width provides per step capacity: how much information each layer can hold and process. Once you have enough refinement steps, additional steps don't help; the bottleneck shifts to how much each step can do.
**Modern Llama 70B** uses `d_model = 8192` with 80 layers. That is wider than the 2017 transformer ratios would suggest for the same parameter count, reflecting the wider not deeper finding from years of scaling law sweeps.
**At a fixed parameter budget, scale width past the depth sweet spot.** The depth threshold scales sub-linearly with parameter count: ~32 layers at 7B, ~80 at 70B, ~126 at 405B.
**The empirical evidence.** Levine et al. (2020, 'Limits to Depth Efficiencies') showed analytically and empirically that past a certain depth, additional layers contribute less than the equivalent width increase. Chinchilla era follow-ups confirmed this when training compute became the primary constraint. The Kaplan et al. and Hoffmann et al. scaling law papers calibrated the relationship between parameters, data, and compute, and recent industrial sweeps (Meta's Llama 3 paper, in particular) reported aspect ratios that match the wider not deeper prescription closely.
**The intuition: composition vs capacity.** Depth provides representational composition: each layer refines features built by previous layers. Width provides per step capacity, how much information each layer can encode. At small parameter budgets, composition is the bottleneck; adding layers gives the model more refinement steps and quality scales cleanly. At large parameter budgets, capacity per step becomes the bottleneck, and the marginal parameter buys more quality if spent on `d_model`.
**Modern frontier shapes.** Llama 3.1 70B: 80 layers, `d_model = 8192`, aspect ratio `d_model / n_layers ≈ 102`. Llama 3.1 405B: 126 layers, `d_model = 16384`, aspect ratio ≈ 130. The aspect ratio settles in the 100-130 range at frontier scale, well above the 2017 Vaswani ratio of `512 / 6 ≈ 85`.
**Caveats.** Depth still matters for tasks requiring long reasoning chains; recent test time compute work (o1 style) leans on depth of reasoning rather than depth of layers but the analogy is informative. Tiny models (sub-1B) sometimes benefit from being deeper than the scaling law fit predicts because composition matters more at limited capacity.
Quick reference
TL;DR
Depth provides composition (refinement steps), width provides per step capacity. Past a depth threshold (~32 for 7B, ~80 for 70B), width scales cleaner; spend the marginal parameter on `d_model`.
Imagine you have a budget to build a hotel. You can either build it taller (more floors) or wider (bigger floors). At first, adding floors helps a lot: each new floor lets guests have a different kind of room. Past a certain height the elevators get slow and adding more floors helps less. At that point, your money is better spent making each floor bigger so each room can fit more stuff. LLMs are the same. Depth (number of layers) is like floors: each layer refines the model's understanding step by step. Width (`d_model`) is like the size of each floor; bigger floors hold more knowledge per layer. Once you have around 32 layers in a small model or 80 in a big one, adding more layers stops helping much and you should grow each layer instead.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Designing a tiny on device model under 1B parameters Small models often benefit from going deeper than the scaling law fit predicts; composition matters more when per step capacity is already limited.
Picking architecture for an MoE model like Llama 4 Maverick Total parameters are dominated by experts, not the dense stack; depth vs width applies to the dense part, but the bigger lever is expert count and routing.
Optimizing for inference time reasoning quality Reasoning models lean on chain-of-thought (search) more than depth of layers; pure depth scaling has weaker returns than test time compute.
Quick scan · 2–4 min
Everything important, quickly.
8 min: define composition vs capacity, walk the empirical anchors at 7B / 70B / 405B, cite Levine and Chinchilla, explain the aspect ratio asymptote, and finish with the cases where the rule breaks.
Side-by-side
Model
Layers
d_model
Aspect (d_model / layers)
Vaswani 2017 base
6
512
85
Llama 3.1 8B
32
4096
128
Llama 3.1 70B
80
8192
102
Llama 3.1 405B
126
16384
130
Mistral Large 3
~88
~9216
~105
Real-world examples
Real products, models, and research that use this idea.
2Llama 3.1 70B: 80 layers, d_model 8192, FFN hidden 28672; aspect ratio 102; slightly squatter than 8B, confirming wider not deeper.
3Llama 3.1 405B: 126 layers, d_model 16384, FFN hidden 53248; aspect ratio 130; width keeps growing faster than depth.
4Llama 4 Maverick (MoE 400B/17B active): the dense stack stays modest depth, with capacity coming from 128 experts; depth vs width applies to the dense part only.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the aspect ratio settle in the 100-130 range instead of continuing to drop?
Sign in or go Pro to see the approach for this follow-up.
QHow does Chinchilla's data vs parameters finding interact with depth vs width?
Sign in or go Pro to see the approach for this follow-up.
QWhat does the depth vs width tradeoff look like for an MoE model?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Believing depth always helps. Past the compute optimal depth for a given parameter budget, additional layers actually hurt; they consume parameters that would have given more quality if spent on width.