Your team debates 32K vs 128K vs 256K vocab. What is the core tradeoff they should frame?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

Shorter sequences versus fatter model edges. A bigger vocabulary compresses text into fewer tokens, cutting attention cost (O(n squared)) and KV cache size. But the embedding matrix and lm_head both scale linearly with vocab_size, so doubling the vocabulary roughly doubles those parameter counts, slows the softmax, and creates rare tokens that may be under-trained. At 32K (Mistral, Llama 2) sequences are long but embeddings are compact. At 128K (Llama 3) sequences shrink and multilingual fertility improves at 4x embedding cost. At 256K (Gemma 2) you get the best compression but more tail risk. The 2026 sweet spot is 100K to 200K.

Your answer

The core tradeoff your team should frame is **shorter sequences versus fatter model edges**. Each of the three vocabulary sizes illustrates a different point on that curve. At 32K (Mistral 7B, Llama 2), the embedding matrix and lm_head stay small, but sequences are long. A paragraph that takes 400 tokens at 32K might compress to only 250 at 128K. Attention is O(n squared) in sequence length, so longer sequences cost significantly more per layer, and the KV cache is proportionally larger. At 128K (Llama 3, Llama 4), sequences shrink meaningfully and multilingual fertility improves because there is room for non-Latin merges. The cost is a 4x larger embedding matrix and lm_head compared to 32K. On a 7B model that adds roughly 786M parameters. At 256K (Gemma 2), you get the best compression and the shortest sequences, but rare tokens in the vocabulary may appear so infrequently in training that their embeddings are under-trained. The softmax over 256K logits is also measurably slower per token. The 2026 consensus is that 100K to 200K is the sweet spot for general LLMs trained on multi-trillion token corpora. Below that, the attention savings go unrealized; above that, rare token problems grow faster than compression gains.

**Vocabulary size** trades sequence length against model parameter cost, and your team should frame the debate around exactly that axis. The upside of a bigger vocabulary is shorter sequences. Each token covers more text, so the same content compresses into fewer positions. Attention is O(n squared) per layer, so halving the sequence cuts attention compute by 4x. KV cache memory scales linearly with sequence length, so shorter sequences also save inference memory. Both numbers matter for long context applications. The downside has three parts. First, the embedding matrix is shaped vocab_size by d_model, and the lm_head is shaped d_model by vocab_size. Doubling vocabulary doubles both of these. On a 7B model with d_model of 4096, going from 32K to 128K adds roughly 786M parameters. Second, the softmax is computed over the full vocabulary every token, adding per-token latency proportional to vocab_size. Third, rare tokens in larger vocabularies may have under-trained embeddings that produce surprising outputs. 2026 production sweet spots: - **tiktoken cl100k_base**: 100K vocab, GPT-4 era. - **tiktoken o200k_base**: 200K vocab, GPT-4o through GPT-5.5. - **Llama 3 and 4**: 128K vocab. - **Mistral 7B family**: 32K vocab. - **Gemma 2**: 256K vocab (notably large for multilingual coverage). The trend across 2023 to 2026 has been clearly upward, driven by longer context windows making attention savings more valuable, multilingual needs requiring vocabulary headroom, and larger training corpora supporting the bigger vocabularies.

Quick reference

TL;DR

Bigger vocab compresses text into shorter sequences (cheaper attention) but fattens the embedding matrix, lm_head, and softmax. The 2026 sweet spot for general LLMs is 100K to 200K.

Memory aid

Easy to grasp

Picture a dictionary you carry around to read every book. A tiny dictionary fits in your pocket but you have to look up every other word, so reading is slow. A huge dictionary lets you read fast because almost every word is one lookup, but the book itself is now twice as heavy in your backpack. Vocabulary size in a language model is the same choice. A small vocabulary breaks text into many small tokens, making the model work through more positions, and the attention math gets expensive. A big vocabulary compresses text into fewer tokens, so the model processes it in fewer steps. But the model also carries one slot per vocabulary entry in two large matrices, so a huge vocabulary makes the model bigger and the final prediction step slower.

Key concepts

Vocabulary Size Embedding Matrix Language Model Head Softmax

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: state the core tradeoff (compression versus parameter cost), walk the three downside components (embedding, lm_head/softmax, under-training), anchor with 2026 production numbers and model examples, and explain why the trend has moved toward larger vocabularies.

Real-world examples

Real products, models, and research that use this idea.

tiktoken cl100k_base ships a 100K vocabulary for GPT-4 era models; o200k_base ships 200K for GPT-4o, o1, o3, GPT-5, and GPT-5.5.
Llama 3 and Llama 4 use a 128K BPE vocabulary, quadrupled from Llama 2's 32K SentencePiece.
Gemma 2 uses a 256K SentencePiece Unigram vocabulary, deliberately large for multilingual headroom.
Mistral 7B and Mistral Large early variants use a 32K SentencePiece BPE vocabulary; later Mistral Large models moved to a larger vocab to match the industry trend.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QDoubling the vocabulary roughly halves sequence length. Why does that not perfectly cancel out the doubled embedding cost?

QHow do production teams decide if their vocabulary is the right size?

QWhy does Gemma use 256K while Mistral models use 32K?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Thinking bigger vocabulary is strictly better. The embedding matrix and lm_head both scale linearly with vocab_size, doubling the vocabulary roughly doubles those parameters and adds real cost to every forward pass.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.