The vocabulary is the fixed set of token ids a tokenizer can emit: a numbered list from 0 to vocab_size minus 1, where each id maps to a specific byte sequence. The size is fixed at tokenizer training time and directly determines the row count of the model's embedding matrix and the column count of the lm_head. Typical 2026 vocab sizes: GPT-4 cl100k_base around 100K, GPT-4o through GPT-5.5 o200k_base around 200K, Llama 3 and Llama 4 at 128K.
**The vocabulary** is the fixed set of token ids a tokenizer can emit. It is a numbered list from 0 to vocab_size minus 1, where each id maps to a specific byte sequence. The model can only ever see and produce ids that are in this list.
The size is fixed at tokenizer training time. For BPE, the training algorithm stops merging when the vocabulary reaches the target size, so vocab_size is a hyperparameter you choose before training. Once set, it is baked into the model.
Vocabulary size matters because two large pieces of the model depend on it directly. The embedding matrix has one row per token id, each row a vector of size hidden_dim. The lm_head (the output projection) has one column per token id, each column another vector of size hidden_dim. Doubling the vocabulary doubles both of these tables.
Typical 2026 sizes. tiktoken's cl100k_base (GPT-4, GPT-3.5-turbo, text-embedding-ada-002) is about 100K tokens. tiktoken's o200k_base (GPT-4o, o1, o3, GPT-5, GPT-5.5) is about 200K tokens. The Llama 3 and Llama 4 tokenizer is 128K tokens. Mistral models use a 32K vocabulary.
A bigger vocabulary means each common string can be represented with fewer tokens (lower fertility), but at the cost of larger embedding and output matrices.
**The vocabulary** of a tokenizer is the fixed, numbered set of all token ids it can produce. It is established at tokenizer training time and never changes during model training or inference. For BPE, the vocabulary is the union of the base alphabet (256 byte values for byte-level BPE) and every symbol created by a merge rule, up to the target vocab size.
The vocabulary size is a load-bearing hyperparameter. Two model components depend on it directly. The token embedding matrix has shape (vocab_size, hidden_dim): one learned vector per token. The output projection (lm_head) has shape (hidden_dim, vocab_size): every forward pass produces a vocab_size-dimensional logit vector per position. Together these two matrices account for a substantial fraction of the parameter count in modern LLMs, especially smaller ones.
Vocab size sits in a tradeoff. Larger vocabularies represent frequent strings with single ids, reducing the number of tokens per word (fertility) and therefore per-token cost. They also reduce sequence length for a given text, which helps attention compute. The cost is parameter count: doubling vocab_size doubles both large vocabulary-aware matrices.
Typical 2026 values. OpenAI's tiktoken cl100k_base is approximately 100,256 tokens (GPT-4, GPT-3.5-turbo, text-embedding-ada-002). o200k_base is approximately 200,019 tokens (GPT-4o, o1, o3, GPT-5, GPT-5.5). Meta's Llama 3 and Llama 4 use a 128K-token byte-level BPE vocabulary, deliberately expanded from Llama 2's 32K SentencePiece vocabulary. Mistral retains a 32K vocabulary. The trend is upward as models scale and multilingual coverage matters more.
Quick reference
TL;DR
A tokenizer vocabulary is the fixed numbered list of token ids the tokenizer can emit. Its size is set at training time and directly determines the row count of the embedding matrix and the column count of the lm_head.
Imagine the language model only knows a finite list of stamps it can use, like a kid with a sticker book. The sticker book is the vocabulary. Every sticker has a number printed on it, and the model can only ever stick stickers it owns. When the tokenizer reads your prompt, it figures out which stickers to lay out to represent your text. When the model replies, it picks stickers one at a time from the same book. The number of pages in the book is the vocabulary size. A bigger book lets the model represent more things with single stickers (so common words become one sticker instead of three), but every page in the book costs memory inside the model. Picking a vocabulary size is picking how big the sticker book should be before you start training.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Picking a very large vocabulary for a small model with limited training data. Large vocabularies have many rare tokens whose embeddings are hard to learn well with limited data; the model wastes capacity on tokens it rarely sees during training.
Trying to add new tokens to a pretrained model's vocabulary without re-training the embedding for those tokens. New token rows are randomly initialized; the model has no learned representation for them and will produce nonsense until fine-tuning establishes their embeddings.
Choosing vocab_size based on context-window intuitions. Vocab size and context window are independent. A 32K-token vocabulary works fine with a 2M-token context window; the two parameters answer different questions.
Quick scan · 2–4 min
Everything important, quickly.
5 min: define vocabulary as the numbered list of ids, identify the embedding and lm_head matrices that depend on its size, walk the tradeoff between fertility and parameter count, cite typical 2026 sizes, and distinguish vocabulary from context window and hidden dimension.
Real-world examples
Real products, models, and research that use this idea.
1OpenAI's tiktoken cl100k_base has 100,256 tokens (used by GPT-4 and GPT-3.5-turbo); o200k_base has approximately 200,019 tokens (used by GPT-4o, o1, o3, GPT-5, GPT-5.5).
2Meta's Llama 3 and Llama 4 use a 128K-token byte-level BPE vocabulary, expanded from Llama 2's 32K SentencePiece vocabulary primarily to improve multilingual and code performance.
3Mistral Large 3 retains a 32K-token SentencePiece-based BPE vocabulary, which contributes to higher fertility on non-English text relative to the larger-vocabulary frontier models.
4Hugging Face tokenizer.json files explicitly list the full vocabulary; you can inspect any open-weight model's tokens by loading the tokenizer and iterating over its vocab.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf you wanted to add a new special token to a pretrained model, what would actually have to happen?
Sign in or go Pro to see the approach for this follow-up.
QWhy did OpenAI move from cl100k_base to o200k_base for GPT-4o and later models?
Sign in or go Pro to see the approach for this follow-up.
QWhat are glitch tokens or unreachable tokens, and why do they exist?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Confusing vocabulary size with context-window size. Vocabulary is how many distinct tokens exist; context window is how many tokens fit in one prompt. They are unrelated parameters.