How does vocabulary size affect embedding table memory footprint at bfloat16 precision, and what are the tradeoffs of very small vs. very large vocabularies?
A model has d_model=4096 and you are considering three vocabulary sizes: 8k, 50k, and 200k tokens. Calculate the bfloat16 memory footprint of the combined embedding table + lm_head for each size. Then describe the practical tradeoffs (sequence length, softmax cost, multilingual coverage) of each vocabulary size choice.
Free · 2 AI evals / day
Vocabulary memory in a transformer lives in two matrices: the embedding table at the input and the `lm_head` at the output. Both have one slot per token, so both have shape `[vocab_size, d_model]`.
bfloat16 stores each number in 2 bytes. So the combined memory is `2 × vocab_size × d_model × 2 bytes`. For `d_model=4096`:
- 8k vocab: about 131 MB
- 50k vocab: about 820 MB
- 200k vocab: about 3.3 GB
Now the tradeoffs. A small vocabulary like 8k splits most words into many pieces, so sequences get long. Longer sequences cost more attention compute and burn through the context window faster, even though the embedding table is tiny.
A large vocabulary like 200k packs text into fewer tokens, which is great for non-English languages and shorter sequences. The price is two-fold: the embedding matrices alone take 3.3 GB, and the output step has to score all 200k tokens every time, which is slow. The middle, around 50k, balances both for English-centric models.
bfloat16 is 2 bytes per value, and the vocabulary lives in two matrices, the embedding and the `lm_head`, each `[vocab_size, d_model]`. So the memory is 2 times vocab_size times d_model times 2 bytes. At d_model 4096, that is about 131 MB at 8k, 820 MB at 50k, and 3.3 GB at 200k.
The tradeoffs run in opposite directions. A small 8k vocabulary fragments text into many tokens, so fertility is high and sequences are long. Since attention scales as `O(n²)`, long sequences are expensive and the context window fills fast. The upside is a tiny embedding table.
A large 200k vocabulary compresses text into fewer tokens, which helps non-English languages reach near-native efficiency and shortens sequences. The cost is real: 3.3 GB of vocabulary weights, plus an output softmax over 200k classes that runs at every decoded token, roughly 4x the work of 50k. The 50k middle is the English sweet spot, while modern multilingual models push to 100k-200k.
Quick reference
TL;DR
Vocabulary memory is 2 × vocab × d_model × 2 bytes; at d_model=4096 that is 131 MB at 8k, 820 MB at 50k, 3.3 GB at 200k, trading sequence length against softmax and memory cost.
Picture each token getting its own labeled box in two warehouses, one where the model reads input and one where it writes output. More tokens means more boxes, so both warehouses get bigger and pricier to keep.
But more boxes also let you pack a long word into a single box instead of spelling it out across five small ones. So a big warehouse stores text in fewer, fatter boxes, while a tiny warehouse forces long chains of little boxes for the same sentence. The choice is a balance: pay for warehouse space with a large vocabulary, or pay for longer chains of boxes with a small one.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Picking a very small vocabulary like 8k for a multilingual English-plus-CJK model High fertility on non-Latin scripts inflates sequence length and the quadratic attention cost far past the memory saved.
Pushing to a 200k vocabulary for an edge model that is memory-constrained and English-only The 3.3 GB interface and the heavier output softmax buy multilingual coverage the deployment never uses.
Applying NLP-scale vocabularies to a tiny-alphabet domain such as DNA or protein sequences The productive token space is only a few thousand patterns, so most embedding rows would never receive gradient and waste memory.
Real products, models, and research that use this idea.
1GPT-5.5 ships o200k_base with roughly 200k tokens, spending several GB on vocabulary weights to cut non-English token counts and per-call cost.
2Llama 3 grew its vocabulary from an earlier 32k to 128k, lifting interface memory from about 524 MB to 2.1 GB while improving code and multilingual compression.
3DNABERT-2 uses a roughly 4k vocabulary because DNA's four-letter alphabet yields at most 4^6 useful 6-mers, so a larger table would sit mostly empty.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QAt 1,000 requests per second with a 200k-vocab model at d_model=4096, how does lm_head matmul cost compare to attention for a 512-token sequence?
Sign in or go Pro to see the approach for this follow-up.
QHow would you allocate a 100k vocabulary budget for a code assistant spanning 50 programming and 20 natural languages?
Sign in or go Pro to see the approach for this follow-up.
QWhen does the sequence-length saving from a larger vocabulary fully pay for its extra softmax and memory cost?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Counting only the embedding table and computing half the real memory, or using 4 bytes when modern LLMs serve at bfloat16's 2 bytes per value.