What is d_model, and which three parts of the architecture scale with it?

Flashcard·Easy·4.0 · 0·~30s·Asked atAi4bharat Modal Labs Ola·Relevant atMistral AI

Attempt it

d_model is the hidden dimension that the residual stream carries from block to block (also called hidden_size or embedding_dim). Three things scale with it: (1) the four attention projections, each d_model x d_model; (2) the FFN hidden dim, ~4 * d_model in ReLU FFNs and ~2.67 * d_model in SwiGLU FFNs; (3) the token embedding table, which is vocab_size x d_model. Every width-scaling decision flows from d_model.

Your answer

**d_model is the width of the residual stream.** It is the dimensionality of the vector that the model carries from block to block, one vector per token per layer. It is also called `hidden_size` (Hugging Face configs). For a 7B-class LLM, d_model is typically 4096. For a 70B-class LLM, it is 8192. **Three architectural quantities that scale with d_model:** 1. **Attention projections.** The Q, K, V, and O matrices are all `d_model x d_model` for full MHA (multi-head attention). Together they hold 4 * d_model^2 parameters per block. 2. **FFN hidden dimension.** The FFN (feed-forward network) expands the d_model vector to a wider d_ff intermediate, then projects back. The hidden dim is typically 4 * d_model in classic ReLU FFNs or ~2.67 * d_model in SwiGLU FFNs. 3. **Token embedding table.** The input embedding matrix is `vocab_size x d_model`. For vocab = 128k and d_model = 8192, that is ~1B parameters. **Other scaling targets:** - **Per head attention dimension**: d_head = d_model / n_heads. For d_model = 4096 with 32 heads, d_head = 128. - **KV cache memory at inference**: 2 * batch * seq_len * n_kv_heads * d_head, which depends on d_model through d_head.

**d_model definition**: the width of the residual stream that flows from block to block. Each token at each layer is represented by a d_model-dim vector. **Three principal scaling targets:** 1. **The four attention projections** (Q, K, V, O). Each is `d_model x d_model` for full MHA. Total attention params per block: 4 * d_model^2. With GQA or MQA, the K and V projections shrink. The Q and O projections remain d_model x d_model. 2. **The FFN hidden dimension** d_ff. In a 2-matrix ReLU FFN, d_ff = 4 * d_model (Vaswani 2017 default). In a 3-matrix SwiGLU FFN, d_ff drops to ~8/3 * d_model to hold block parameter count constant. The FFN matrices are `d_model x d_ff` and `d_ff x d_model`. 3. **The embedding table** (input embedding and the untied unembedding). Shape `vocab_size x d_model`. For vocab sizes of 32k to 256k, this is a multi hundred million to multi-billion parameter component. **Why d_model is the master knob.** Most other dimensions are tied multiples of d_model. If you widen d_model by 2x, you scale attention params by 4x, FFN params by 4x, and the embedding table by 2x. The total block parameter count is `12 * d_model^2` (dense Vaswani-style block), so block params scale quadratically with d_model. **Other quantities that depend on d_model.** - d_head = d_model / n_heads (per head attention dimension). - KV cache memory at decode: 2 * batch * seq_len * n_kv_heads * d_head * 2 bytes per layer in bf16. It depends on d_model through d_head.

TL;DR

d_model is the hidden size of the residual stream. Attention head dim, FFN hidden dim, KV cache, and the embedding table all scale with d_model.

Memory aid

Easy to grasp

Picture the transformer as a highway with a fixed number of lanes. d_model is how many lanes the highway has. A wider highway carries more cars per cross-section, so more bridges must be wider too. The bridges in this picture are the attention machinery, the per-word processing layer and the word to numbers lookup phone book. The last bridge is the running notebook of past attention values that decoders carry around at answer time. Every one of them scales with the lane count. Pick d_model = 4096 for a 7B-class model, d_model = 8192 for a 70B-class model, and every other width number in the architecture follows.

Key concepts

d_model (hidden size)Embeddings Feed-forward network

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

d_model is the single most referenced constant in transformer code. Every config file lists it. Every parameter-counting calculation starts from it. Every scaling-law analysis parameterizes width through it.

What d_model means

d_model is the dimensionality of the residual stream, which is the working memory buffer that flows from block to block in a transformer. At every layer and every sequence position, the model carries a d_model-dimensional vector representing that token's running state.

Other names for the same quantity:

hidden_size (Hugging Face configs).
embedding_dim (PyTorch tutorials, especially for input embeddings).
n_embd (the original GPT-2 codebase).
Just 'the hidden size' in casual discussion.

It is a deliberate architectural choice, set in the config file before training begins. It does not change layer to layer. The residual stream width is constant throughout the stack. This is structurally important, since residual connections require the input and output of each sublayer to have the same shape.

Typical values:

768 to 1024 for small models (GPT-2 small, BERT-base).
4096 for 7B-class LLMs (Llama, Mistral, Qwen, DeepSeek small).
8192 for 70B-class LLMs (Llama 3 70B).
12288+ for very large models (GPT-3 175B).

The four scaling targets

Why d_model is 'the' width knob

Inference cost consequence: KV cache memory

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Quantity	Formula	Llama 3 70B value
d_model	(set directly)	8192
Attention Q/O matrix	d_model x d_model	8192 x 8192
Attention K/V matrix (GQA 8x)	d_model x d_model/8	8192 x 1024
FFN hidden d_ff	3.5 * d_model	28672
FFN matrix shape	d_model x d_ff	8192 x 28672
Per-head attention d_head	d_model / n_heads	8192 / 64 = 128
Embedding table	vocab_size x d_model	128256 x 8192
KV cache per token (bf16)	2 * n_layers * n_kv_heads * d_head * 2	320 KB

Real products, models, and research that use this idea.

Llama 3 8B: d_model = 4096. d_ff = 14336 (3.5x). vocab = 128256. Embedding table: 525M params. Per-head dim: 128 (32 heads). Block param: ~218M.
Llama 3 70B: d_model = 8192. d_ff = 28672 (3.5x). vocab = 128256. Embedding table: 1.05B params. Per-head dim: 128 (64 heads). Block param: ~855M.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is d_head almost universally 128 across modern open-weight LLMs?

128 maps well onto GPU tensor-core tiles, and FlashAttention-style kernels are most tuned for it. Smaller d_head wastes capacity per head; larger d_head (FlashAttention supports up to 256, which Gemma uses) costs more memory and compute per head. Once d_head = 128 was confirmed empirically as a quality-respecting choice, most of the field standardized on it. Llama 1/2/3/4, Mistral, Qwen, DeepSeek all use d_head = 128. This is a hardware-driven architectural convergence.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Confusing d_model with the FFN hidden dim or with the per-head attention dim. d_model is the *residual stream* width; the FFN hidden dim is a multiple of it; the per-head dim is d_model divided by the number of heads.

60 second bullets to scan on the way to the call.

d_model = width of the residual stream
Attention Q, K, V, O scale with d_model (matrices of size d_model x d_model in full MHA)

Primary sources. Browse if you want the original framing.