Your team debates 32K vs 128K vs 256K vocab. What is the core tradeoff they should frame?

The core tradeoff your team should frame is **shorter sequences versus fatter model edges**. Each of the three vocabulary sizes illustrates a different point on that curve. At 32K (Mistral 7B, Llama 2), the embedding matrix and lm_head stay small, but sequences are long. A paragraph that takes 400 tokens at 32K might compress to only 250 at 128K. Attention is O(n squared) in sequence length, so longer sequences cost significantly more per layer, and the KV cache is proportionally larger. At 128K (Llama 3, Llama 4), sequences shrink meaningfully and multilingual fertility improves because there is room for non-Latin merges. The cost is a 4x larger embedding matrix and lm_head compared to 32K. On a 7B model that adds roughly 786M parameters. At 256K (Gemma 2), you get the best compression and the shortest sequences, but rare tokens in the vocabulary may appear so infrequently in training that their embeddings are under-trained. The softmax over 256K logits is also measurably slower per token. The 2026 consensus is that 100K to 200K is the sweet spot for general LLMs trained on multi-trillion token corpora. Below that, the attention savings go unrealized; above that, rare token problems grow faster than compression gains.

**Vocabulary size** trades sequence length against model parameter cost, and your team should frame the debate around exactly that axis. The upside of a bigger vocabulary is shorter sequences. Each token covers more text, so the same content compresses into fewer positions. Attention is O(n squared) per layer, so halving the sequence cuts attention compute by 4x. KV cache memory scales linearly with sequence length, so shorter sequences also save inference memory. Both numbers matter for long context applications. The downside has three parts. First, the embedding matrix is shaped vocab_size by d_model, and the lm_head is shaped d_model by vocab_size. Doubling vocabulary doubles both of these. On a 7B model with d_model of 4096, going from 32K to 128K adds roughly 786M parameters. Second, the softmax is computed over the full vocabulary every token, adding per-token latency proportional to vocab_size. Third, rare tokens in larger vocabularies may have under-trained embeddings that produce surprising outputs. 2026 production sweet spots: - **tiktoken cl100k_base**: 100K vocab, GPT-4 era. - **tiktoken o200k_base**: 200K vocab, GPT-4o through GPT-5.5. - **Llama 3 and 4**: 128K vocab. - **Mistral 7B family**: 32K vocab. - **Gemma 2**: 256K vocab (notably large for multilingual coverage). The trend across 2023 to 2026 has been clearly upward, driven by longer context windows making attention savings more valuable, multilingual needs requiring vocabulary headroom, and larger training corpora supporting the bigger vocabularies.