A model has d_model=4096 and you are considering three vocabulary sizes: 8k, 50k, and 200k tokens. Calculate the bfloat16 memory footprint of the combined embedding table + lm_head for each size. Then describe the practical tradeoffs (sequence length, softmax cost, multilingual coverage) of each vocabulary size choice.
Vocabulary memory is 2 × vocab × d_model × 2 bytes; at d_model=4096 that is 131 MB at 8k, 820 MB at 50k, 3.3 GB at 200k, trading sequence length against softmax and memory cost.
Picture each token getting its own labeled box in two warehouses, one where the model reads input and one where it writes output. More tokens means more boxes, so both warehouses get bigger and pricier to keep. But more boxes also let you pack a long word into a single box instead of spelling it out across five small ones. So a big warehouse stores text in fewer, fatter boxes, while a tiny warehouse forces long chains of little boxes for the same sentence. The choice is a balance: pay for warehouse space with a large vocabulary, or pay for longer chains of boxes with a small one.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Vocabulary size is one of the few tokenizer decisions that simultaneously touches memory, decode compute, and sequence length, which is why it shows up in senior LLM interviews. A candidate who can move from a clean byte formula to the downstream tradeoffs demonstrates the kind of end to end reasoning that architecture decisions actually require.
The trap is to treat this as pure arithmetic. The numbers are easy once you remember the two matrices and the byte width. The harder, more valuable part is explaining why a small vocabulary and a large vocabulary each cost something, in different currencies, and how to pick based on the bottleneck you actually have.
The phrasing of the question is a hint in itself. It asks for a calculation and then for tradeoffs, so a complete answer does both: it produces the three memory figures cleanly, then explains what each vocabulary choice buys and costs. We will derive the memory figures, then trace the sequence-length tax of small vocabularies, the softmax tax of large ones, the multilingual coverage argument that pushes modern models toward 100k-200k, and the weight-tying factor that silently halves every number.
The memory arithmetic, step by step
The vocabulary lives in two matrices: the embedding table [vocab_size, d_model] at the input and the lm_head [d_model, vocab_size] at the output. bfloat16 stores each value in 2 bytes, which gives the formula:
The leading 2 counts both matrices; the trailing 2 is the byte width.
At d_model=4096: an 8k vocabulary costs 2 × 8,000 × 4,096 × 2 = 131 MB. A 50k vocabulary costs 820 MB. A 200k vocabulary costs 3.3 GB. Two common slips change these numbers: forgetting the lm_head halves them, and assuming float32 doubles them. If the model uses weight tying, the embedding and head share one tensor, so you divide by 2 across the board.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
def vocab_memory_gb(vocab_size: int, d_model: int,
weight_tied: bool = False,
bytes_per_param: int = 2) -> float:
"""Vocabulary-related memory in GB; bfloat16 is 2 bytes per value."""
n_matrices = 1 if weight_tied else 2 # embedding + lm_head unless tied
total_params = n_matrices * vocab_size * d_model
return total_params * bytes_per_param / (1024 ** 3)
d_model = 4096
for vocab_size in [8_000, 50_000, 200_000]:
untied = vocab_memory_gb(vocab_size, d_model)
tied = vocab_memory_gb(vocab_size, d_model, weight_tied=True)
print(f"{vocab_size:>7,}: {untied:5.2f} GB untied, {tied:5.2f} GB tied")
# 8,000: 0.12 GB untied, 0.06 GB tied
# 50,000: 0.76 GB untied, 0.38 GB tied
# 200,000: 3.05 GB untied, 1.53 GB tied| Concern | Small vocab (8k) | Large vocab (200k) |
|---|---|---|
| Interface memory at d=4096 | ~131 MB | ~3.3 GB |
| Sequence length / fertility | High (many tokens per word) | Low (compact tokens) |
| Attention cost | Worse (longer sequences, O(n squared)) | Better (shorter sequences) |
| Output softmax cost | Cheap per token | Expensive per token |
| Multilingual coverage | Poor outside English | Near-native across scripts |
| Best fit | Tiny-alphabet domains | Multilingual frontier models |
Real products, models, and research that use this idea.
- GPT-5.5 ships o200k_base with roughly 200k tokens, spending several GB on vocabulary weights to cut non-English token counts and per-call cost.
- Llama 3 grew its vocabulary from an earlier 32k to 128k, lifting interface memory from about 524 MB to 2.1 GB while improving code and multilingual compression.
What an interviewer would ask next. Try answering before peeking at the approach.
QAt 1,000 requests per second with a 200k-vocab model at d_model=4096, how does lm_head matmul cost compare to attention for a 512-token sequence?
Sketch lm_head FLOPs as 2 × d_model × vocab per token against per-token attention FLOPs, and note the head can dominate at large vocabularies.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Counting only the embedding table and computing half the real memory, or using 4 bytes when modern LLMs serve at bfloat16's 2 bytes per value.
60 second bullets to scan on the way to the call.
The bytes formula for vocabulary memory at bfloat16
Why two matrices, not one, enter the count
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.