In a transformer, exactly two components grow as you add more tokens to the vocabulary.
The first is the **embedding matrix** at the start of the model. It is a table with one row for every token, and each row is a vector of size `d_model`. To process an input token, the model looks up its row by token ID. Its shape is `[vocab_size, d_model]`, so adding tokens adds rows.
The second is the `lm_head` at the end. It is a linear layer that turns the model's internal hidden state, of size `d_model`, into one score for every possible next token. Its shape is `[d_model, vocab_size]`, so adding tokens adds columns.
Everything in between, the attention layers and the feed-forward layers, is sized by `d_model` and the number of layers only. Those layers process fixed-size vectors and have no idea how big the vocabulary is. So the right answer pairs the embedding table with the output head, and rules out attention and feed-forward weights entirely.
Exactly two tensors scale with **vocabulary size**: the input embedding matrix, shape `[vocab_size, d_model]`, and the output `lm_head`, shape `[d_model, vocab_size]`. Together they hold 2 times vocab_size times d_model parameters, both linear in the vocabulary.
Plug in real numbers. For d_model 4096 and a 100k vocabulary, that is 2 times 100,000 times 4096, about 820M parameters. At bfloat16, that is roughly 1.6 GB of weights devoted purely to the vocabulary interface, a meaningful slice of a 7B model.
Everything else is vocabulary-agnostic. The attention Q, K, V, and output projections are d_model by d_model. The feed-forward layers depend on d_model and the hidden width. Layer norms are d_model vectors. None of them carry a vocab_size term. The trap answers attach scaling to attention heads or the feed-forward block, which is exactly the misconception this question probes.
Quick reference
TL;DR
Only the input embedding [vocab_size × d_model] and the output lm_head [d_model × vocab_size] scale with vocabulary size; attention and feed-forward weights depend on d_model alone.
Think of a transformer as a translation office. At the front door sits a filing cabinet with one drawer per word the office knows, and looking up a word means pulling its drawer. That front cabinet is how the office turns a word into notes it can work with.
At the back door sits a second cabinet, also one slot per known word, used to decide which word to write next. Add more words to the office's knowledge and both cabinets grow, one drawer at a time. But the clerks in the middle, who shuffle and combine the files, do the same job no matter how many words exist. They never get bigger when the vocabulary does.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Estimating only the embedding table while ignoring the untied output head Counting one matrix halves the true vocabulary cost and underestimates memory by a full embedding table's worth.
Reasoning about a model that uses weight tying with the two-matrix formula Tied weights share one tensor, so the cost is vocab_size times d_model, not twice that.
Predicting attention or feed-forward memory growth from vocabulary changes Those layers depend on d_model and depth alone, so a vocabulary change leaves their size untouched.
Quick scan · 2–4 min
Everything important, quickly.
2 min: embedding [V×d] + head [d×V] + 2·V·d formula + backbone is vocab-agnostic + weight tying + output softmax cost at large vocab.
Real-world examples
Real products, models, and research that use this idea.
1Llama 3 expanded its vocabulary from an earlier 32k to 128k, quadrupling the embedding plus head parameters at d_model=4096 in exchange for shorter sequences on code and multilingual text.
2GPT-5.5 ships the o200k_base tokenizer with roughly 200k tokens, paying several GB of dedicated interface weights to cut non-English token counts.
3GPT-2 used weight tying to share its embedding and output head, a standard trick that halves vocabulary-related memory in smaller models.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QFor a 7B model with d_model=4096, what fraction of parameters is vocabulary-related at 32k vs 128k vocab, untied?
Sign in or go Pro to see the approach for this follow-up.
QWhy does the output softmax over a large vocabulary not benefit from FlashAttention, and what reduces its cost?
Sign in or go Pro to see the approach for this follow-up.
QWhen would you choose to keep the embedding and head untied despite the memory cost?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Attributing vocabulary scaling to attention heads or feed-forward layers. Those depend only on d_model and the layer count; vocabulary size never appears in them.