Compare hidden_size, d_model, and embedding_dim, three names for the same thing or three different things?

Flashcard·Easy·4.0 · 0·~30s·Asked atMercor Persistent Shield Ai·Relevant atMicrosoft

Attempt it

Same thing. All three name the per-token vector width that flows through the residual stream from one transformer block to the next. d_model is the term used in the original Vaswani 2017 paper and most academic literature. hidden_size is the Hugging Face config.json key (e.g. hidden_size: 4096 for Llama-2 7B). embedding_dim is the PyTorch nn.Embedding constructor argument that builds the token embedding table. They are always equal because the embedding output feeds directly into the residual stream and every block reads/writes vectors of this same width.

Your answer

**They all refer to the same single number**: the per-token vector width that flows through the residual stream from one transformer block to the next. The three names come from three different communities: - **d_model**: the term used in the original 'Attention Is All You Need' paper (Vaswani et al. 2017) and most academic literature about transformers. - **hidden_size**: the field name in Hugging Face's `config.json` (e.g. `hidden_size: 4096` for Llama-2 7B). - **embedding_dim**: the PyTorch `nn.Embedding` constructor argument that builds the token embedding lookup table. The reason they have to be equal is structural. The token embedding table outputs a vector of width `embedding_dim` per token. That vector enters the residual stream, which carries vectors of width `d_model` through every transformer block. Each attention and FFN block reads and writes vectors of the residual stream's width. If any of the three values disagreed, the model could not be built. The embedding output would not fit into the residual stream, or the attention input would not match what came out of the previous block. PyTorch would error at construction time. **Worked example**: in Llama-2 7B, `hidden_size = 4096`, `embedding_dim = 4096`, and you would write `d_model = 4096` if you were describing the model in a paper. One number, three labels.

**Three names, one number.** All three (`d_model`, `hidden_size`, `embedding_dim`) refer to the per-token vector width that flows through the residual stream of a transformer. The naming comes from three different traditions: - **d_model**: Vaswani et al. 2017 introduced this notation. Academic papers continue using it. - **hidden_size**: the Hugging Face `config.json` standard. Llama-2 7B has `hidden_size: 4096`; GPT-2 small has `hidden_size: 768`. - **embedding_dim**: PyTorch's `nn.Embedding(num_embeddings, embedding_dim)` parameter. The three values are always equal because of how transformers are built: - The token embedding lookup produces vectors of width `embedding_dim`. - The residual stream carries vectors of width `d_model` through every block. - Each block (attention, FFN) reads `d_model` and writes `d_model`. - The final output projection consumes `d_model` and produces vocab-size logits. If the three values disagreed, the model could not be built; PyTorch would raise a shape-mismatch error at construction time. Other fields in the config that interact with this number include `num_attention_heads` (must divide `hidden_size`) and `intermediate_size` (typically 4 * hidden_size for plain FFN, or 8/3 * hidden_size for SwiGLU). All of these are derived or constrained relative to the one shared residual-stream width. The practical takeaway: when reading any transformer codebase, treat `d_model`, `hidden_size`, `embedding_dim`, and occasionally `hidden_dim` or `model_dim`, as synonyms for one quantity. The naming variation is purely historical.

TL;DR

Three names, one number. d_model (papers), hidden_size (HF config), embedding_dim (PyTorch nn.Embedding) all refer to the per-token residual-stream width.

Memory aid

Easy to grasp

Imagine three different stores selling the exact same product but with different labels: one calls it a 'large coffee', another 'a 16-ounce', the third 'one of the bigger sizes'. The drink is the same drink. d_model, hidden_size, and embedding_dim are like that, the same vector width, just named differently depending on whether you are reading a research paper, a Hugging Face config file, or PyTorch source code. The number behind all three labels is one quantity: how wide each token's vector is as it flows through the transformer.

Key concepts

d_model Transformer architecture Residual stream Token embedding dimension

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

d_model, hidden_size, and embedding_dim are three names for the same number: the per-token vector width that flows through the residual stream of a transformer. The naming variation is purely a function of which community produced the term, and the values are always equal because the architecture forces them to be.

Missing this is a small but revealing interview signal: it tells the interviewer whether a candidate has actually built a transformer or only read about one. The first time you implement attention, you discover that the embedding output, the residual stream, and every block's input must all be the same shape, and the names converge in your head. This deep dive walks each name and where it lives, why the residual stream forces them to be equal, how this number relates to other 'dim' fields you might confuse it with, and the practical guide for reading any transformer config or codebase.

Where each name comes from

Three names from three different traditions, all pointing at the same quantity.

d_model: the academic paper notation

Vaswani et al. 2017 introduced d_model as the symbol for the dimensionality of the model's internal representations. The paper used it consistently across formulas, figures, and ablation tables. Subsequent transformer-related papers (BERT, GPT, T5, Llama) maintained the convention.

When you read a paper about transformers, scaling laws, attention variants, or interpretability, expect to see d_model or sometimes its variants d_{model}, D, or just d if context is clear.

hidden_size: the Hugging Face config standard

Hugging Face's transformers library, the de facto standard interface for working with pretrained models, uses hidden_size in every model's config.json. The field name reflects the broader 'hidden state' terminology from RNN-era deep learning, where the hidden state width was a primary hyperparameter.

Looking at a few examples:

Llama-2 7B: hidden_size: 4096
GPT-2 small: hidden_size: 768
BERT base: hidden_size: 768
Mistral 7B: hidden_size: 4096

The field name is consistent across model families. If you need to know a model's d_model, look up hidden_size in its config.

embedding_dim: the PyTorch nn.Embedding argument

PyTorch's nn.Embedding(num_embeddings, embedding_dim) constructor takes the vocabulary size and the output dimensionality. The second argument names the per-token embedding vector width:

python

import torch.nn as nn

embedding = nn.Embedding(num_embeddings=32000, embedding_dim=4096)

This builds an embedding matrix of shape (32000, 4096). The 4096 is the d_model of the model that uses this embedding.

Within PyTorch source code itself you also see embed_dim, d_model, and occasionally model_dim referring to the same quantity in different modules. The library is not entirely consistent internally, but the values are.

Other names you might encounter

n_embd: GPT-2's original Hugging Face config used this. Modern Llama-family configs use hidden_size instead.
model_dim or hidden_dim: occasional alternatives in research code.
width: scaling-law papers sometimes call the residual stream width simply 'width' in contrast to depth (number of layers).

All of these are synonyms for d_model.

If you see a number used as the per-token vector width anywhere in the transformer pipeline, it is d_model regardless of what field or variable name it lives under.

Why the residual stream forces them all to be equal

How d_model relates to head_dim

How d_model relates to intermediate_size and other dimensions

Practical reading guide and the broader lesson

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Llama-2 7B config.json: hidden_size = 4096. Same value as d_model in any paper about the model and as embedding_dim in any PyTorch implementation.
GPT-2 small: hidden_size = 768, the canonical small-transformer reference value.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the residual stream width have to stay constant through every block of the model?

Residual connections add the block output to the block input. For the addition to be well-defined, the two tensors must have the same shape. This forces every transformer block (attention, FFN) to read and write tensors of the same trailing dimensionality. The architecture cannot taper or expand the residual stream mid-model without breaking the residual addition, which is structural to how transformers train.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating these as three different hyperparameters you can set independently. They are always the same value, the residual stream width must match the embedding output which must match every block's input.

60 second bullets to scan on the way to the call.

The three names and which communities use which
Why the residual stream width forces the three values to be equal

Primary sources. Browse if you want the original framing.