Define d_model and name three architecture knobs that scale with it

**d_model is the width of the residual stream.** It is the dimensionality of the vector that the model carries from block to block, one vector per token per layer. It is also called `hidden_size` (Hugging Face configs), `embedding_dim` (PyTorch tutorials), or simply 'the hidden size'. All three names refer to the same quantity. For a 7B-class LLM, d_model is typically 4096. For a 70B-class LLM, it is 8192. For frontier models in the hundreds of billions, it climbs to 12288 or higher. **Three architectural quantities that scale with d_model:** 1. **Attention projections.** The Q, K, V, and O matrices in each attention sublayer are all `d_model x d_model` for full MHA. Each one has d_model^2 parameters; four of them total 4 * d_model^2 parameters per block. 2. **FFN hidden dimension.** The FFN expands the d_model vector to a wider d_ff intermediate, then projects back. The hidden dim is typically 4 * d_model in classic ReLU FFNs (Vaswani 2017) or ~2.67 * d_model in SwiGLU FFNs (Llama, Mistral, modern). Every choice of FFN width is expressed as a multiple of d_model. 3. **Token embedding table.** The input embedding matrix is `vocab_size x d_model`. Vocab is typically 32k to 256k. For vocab = 128k and d_model = 8192, that is ~1B parameters in the embedding table alone. **Bonus scaling targets** (not in the top three but worth knowing): - **Per-head attention dimension**: d_head = d_model / n_heads. For d_model = 4096 with 32 heads, d_head = 128. - **KV cache memory at inference**: 2 * batch * seq_len * n_kv_heads * d_head, which depends on d_model through d_head. - **Layer norm / RMSNorm scale parameters**: one learned scalar per channel, d_model channels per norm.

**d_model definition**: the width of the residual stream that flows from block to block. Each token at each layer is represented by a d_model-dim vector. It is the single most-referenced constant in transformer code and the primary 'width' knob in scaling laws. **Three principal scaling targets:** 1. **The four attention projections** (Q, K, V, O). Each is `d_model x d_model` for full MHA. Total attention params per block: 4 * d_model^2. With GQA or MQA, the K and V projections shrink (k_head and v_head dims become d_model / GQA_ratio), but the Q and O projections remain d_model x d_model. 2. **The FFN hidden dimension** d_ff. In a 2-matrix ReLU FFN, d_ff = 4 * d_model (Vaswani 2017 default). In a 3-matrix SwiGLU FFN, d_ff drops to ~8/3 * d_model ~= 2.67 * d_model to hold block parameter count constant. The FFN matrices are `d_model x d_ff` and `d_ff x d_model`. 3. **The embedding table** (input embedding and the untied unembedding). Shape `vocab_size x d_model`. For modern vocab sizes (32k for Llama 2, 128k for Llama 3 and Mistral, 256k for Gemma), this is a multi hundred million to multi-billion parameter component. **Why d_model is the master knob.** Scaling laws (Kaplan 2020, Chinchilla 2022) parameterize model size primarily through d_model: most other dimensions are tied multiples of it. If you widen d_model by 2x, you scale attention params by 4x (quadratic in d_model^2), FFN params by 4x, and the embedding table by 2x. The total block parameter count is `12 * d_model^2` (dense Vaswani-style block), so block params scale quadratically with d_model. This is what makes d_model the dominant width knob: a 2x width increase is a 4x parameter increase in the body of the model. **Other quantities that depend on d_model but are not the top three knobs.** - d_head = d_model / n_heads (per-head attention dimension; n_heads is set independently but d_head follows from d_model). - KV cache memory at decode: 2 * batch * seq_len * n_kv_heads * d_head * 2 bytes