Which part of a transformer model's parameter count scales directly and linearly with vocabulary size?

In a transformer, exactly two components grow as you add more tokens to the vocabulary. The first is the **embedding matrix** at the start of the model. It is a table with one row for every token, and each row is a vector of size `d_model`. To process an input token, the model looks up its row by token ID. Its shape is `[vocab_size, d_model]`, so adding tokens adds rows. The second is the `lm_head` at the end. It is a linear layer that turns the model's internal hidden state, of size `d_model`, into one score for every possible next token. Its shape is `[d_model, vocab_size]`, so adding tokens adds columns. Everything in between, the attention layers and the feed-forward layers, is sized by `d_model` and the number of layers only. Those layers process fixed-size vectors and have no idea how big the vocabulary is. So the right answer pairs the embedding table with the output head, and rules out attention and feed-forward weights entirely.

Exactly two tensors scale with **vocabulary size**: the input embedding matrix, shape `[vocab_size, d_model]`, and the output `lm_head`, shape `[d_model, vocab_size]`. Together they hold 2 times vocab_size times d_model parameters, both linear in the vocabulary. Plug in real numbers. For d_model 4096 and a 100k vocabulary, that is 2 times 100,000 times 4096, about 820M parameters. At bfloat16, that is roughly 1.6 GB of weights devoted purely to the vocabulary interface, a meaningful slice of a 7B model. Everything else is vocabulary-agnostic. The attention Q, K, V, and output projections are d_model by d_model. The feed-forward layers depend on d_model and the hidden width. Layer norms are d_model vectors. None of them carry a vocab_size term. The trap answers attach scaling to attention heads or the feed-forward block, which is exactly the misconception this question probes.