A tokenizer has a 'vocabulary'. What is it, and why does its size matter for the model?

**The vocabulary** is the fixed set of token ids a tokenizer can emit. It is a numbered list from 0 to vocab_size minus 1, where each id maps to a specific byte sequence. The model can only ever see and produce ids that are in this list. The size is fixed at tokenizer training time. For BPE, the training algorithm stops merging when the vocabulary reaches the target size, so vocab_size is a hyperparameter you choose before training. Once set, it is baked into the model. Vocabulary size matters because two large pieces of the model depend on it directly. The embedding matrix has one row per token id, each row a vector of size hidden_dim. The lm_head (the output projection) has one column per token id, each column another vector of size hidden_dim. Doubling the vocabulary doubles both of these tables. Typical 2026 sizes. tiktoken's cl100k_base (GPT-4, GPT-3.5-turbo, text-embedding-ada-002) is about 100K tokens. tiktoken's o200k_base (GPT-4o, o1, o3, GPT-5, GPT-5.5) is about 200K tokens. The Llama 3 and Llama 4 tokenizer is 128K tokens. Mistral models use a 32K vocabulary. A bigger vocabulary means each common string can be represented with fewer tokens (lower fertility), but at the cost of larger embedding and output matrices.

**The vocabulary** of a tokenizer is the fixed, numbered set of all token ids it can produce. It is established at tokenizer training time and never changes during model training or inference. For BPE, the vocabulary is the union of the base alphabet (256 byte values for byte-level BPE) and every symbol created by a merge rule, up to the target vocab size. The vocabulary size is a load-bearing hyperparameter. Two model components depend on it directly. The token embedding matrix has shape (vocab_size, hidden_dim): one learned vector per token. The output projection (lm_head) has shape (hidden_dim, vocab_size): every forward pass produces a vocab_size-dimensional logit vector per position. Together these two matrices account for a substantial fraction of the parameter count in modern LLMs, especially smaller ones. Vocab size sits in a tradeoff. Larger vocabularies represent frequent strings with single ids, reducing the number of tokens per word (fertility) and therefore per-token cost. They also reduce sequence length for a given text, which helps attention compute. The cost is parameter count: doubling vocab_size doubles both large vocabulary-aware matrices. Typical 2026 values. OpenAI's tiktoken cl100k_base is approximately 100,256 tokens (GPT-4, GPT-3.5-turbo, text-embedding-ada-002). o200k_base is approximately 200,019 tokens (GPT-4o, o1, o3, GPT-5, GPT-5.5). Meta's Llama 3 and Llama 4 use a 128K-token byte-level BPE vocabulary, deliberately expanded from Llama 2's 32K SentencePiece vocabulary. Mistral retains a 32K vocabulary. The trend is upward as models scale and multilingual coverage matters more.