Compare hidden_size, d_model, and embedding_dim, three names for the same thing or three different things?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

Same thing. All three name the per-token vector width that flows through the residual stream from one transformer block to the next. d_model is the term used in the original Vaswani 2017 paper and most academic literature. hidden_size is the Hugging Face config.json key (e.g. hidden_size: 4096 for Llama-2 7B). embedding_dim is the PyTorch nn.Embedding constructor argument that builds the token embedding table. They are always equal because the embedding output feeds directly into the residual stream and every block reads/writes vectors of this same width.

Your answer

**They all refer to the same single number**: the per-token vector width that flows through the residual stream from one transformer block to the next. The three names come from three different communities: - **d_model**: the term used in the original 'Attention Is All You Need' paper (Vaswani et al. 2017) and most academic literature about transformers. - **hidden_size**: the field name in Hugging Face's `config.json` (e.g. `hidden_size: 4096` for Llama-2 7B). - **embedding_dim**: the PyTorch `nn.Embedding` constructor argument that builds the token embedding lookup table. The reason they have to be equal is structural. The token embedding table outputs a vector of width `embedding_dim` per token. That vector enters the residual stream, which carries vectors of width `d_model` through every transformer block. Each attention and FFN block reads and writes vectors of the residual stream's width. If any of the three values disagreed, the model could not be built. The embedding output would not fit into the residual stream, or the attention input would not match what came out of the previous block. PyTorch would error at construction time. **Worked example**: in Llama-2 7B, `hidden_size = 4096`, `embedding_dim = 4096`, and you would write `d_model = 4096` if you were describing the model in a paper. One number, three labels.

**Three names, one number.** All three (`d_model`, `hidden_size`, `embedding_dim`) refer to the per-token vector width that flows through the residual stream of a transformer. The naming comes from three different traditions: - **d_model**: Vaswani et al. 2017 introduced this notation. Academic papers continue using it. - **hidden_size**: the Hugging Face `config.json` standard. Llama-2 7B has `hidden_size: 4096`; GPT-2 small has `hidden_size: 768`. - **embedding_dim**: PyTorch's `nn.Embedding(num_embeddings, embedding_dim)` parameter. The three values are always equal because of how transformers are built: - The token embedding lookup produces vectors of width `embedding_dim`. - The residual stream carries vectors of width `d_model` through every block. - Each block (attention, FFN) reads `d_model` and writes `d_model`. - The final output projection consumes `d_model` and produces vocab-size logits. If the three values disagreed, the model could not be built; PyTorch would raise a shape-mismatch error at construction time. Other fields in the config that interact with this number include `num_attention_heads` (must divide `hidden_size`) and `intermediate_size` (typically 4 * hidden_size for plain FFN, or 8/3 * hidden_size for SwiGLU). All of these are derived or constrained relative to the one shared residual-stream width. The practical takeaway: when reading any transformer codebase, treat `d_model`, `hidden_size`, `embedding_dim`, and occasionally `hidden_dim` or `model_dim`, as synonyms for one quantity. The naming variation is purely historical.

Quick reference

TL;DR

Three names, one number. d_model (papers), hidden_size (HF config), embedding_dim (PyTorch nn.Embedding) all refer to the per-token residual-stream width.

Memory aid

Easy to grasp

Imagine three different stores selling the exact same product but with different labels: one calls it a 'large coffee', another 'a 16-ounce', the third 'one of the bigger sizes'. The drink is the same drink. d_model, hidden_size, and embedding_dim are like that, the same vector width, just named differently depending on whether you are reading a research paper, a Hugging Face config file, or PyTorch source code. The number behind all three labels is one quantity: how wide each token's vector is as it flows through the transformer.

Key concepts

d_model Transformer architecture Residual stream Token embedding dimension

Detailed answer & concept explanation

~10 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

60s: state the three names refer to the same residual stream width, name where each is used (papers, HF config, PyTorch), explain why they must be equal (residual addition requires matching shapes), give a worked example (Llama-2 7B: 4096), distinguish from head_dim and intermediate_size which are different quantities.

Real-world examples

Real products, models, and research that use this idea.

Llama-2 7B config.json: hidden_size = 4096. Same value as d_model in any paper about the model and as embedding_dim in any PyTorch implementation.
GPT-2 small: hidden_size = 768, the canonical small-transformer reference value.
Llama-2 70B: hidden_size = 8192, larger residual stream for the larger model.
Mistral 7B: hidden_size = 4096, matches Llama-2 7B for infrastructure reuse.
Llama 4 Maverick and DeepSeek V4 expose hidden_size in their configs as the residual stream width; the value scales with model class.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the residual stream width have to stay constant through every block of the model?

QWhat is the relationship between d_model and head_dim, and why is it different from d_model and intermediate_size?

QWhat happens with weight tying between input embedding and output projection?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating these as three different hyperparameters you can set independently. They are always the same value, the residual stream width must match the embedding output which must match every block's input.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.