A tokenizer has a 'vocabulary'. What is it, and why does its size matter for the model?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

The vocabulary is the fixed set of token ids a tokenizer can emit: a numbered list from 0 to vocab_size minus 1, where each id maps to a specific byte sequence. The size is fixed at tokenizer training time and directly determines the row count of the model's embedding matrix and the column count of the lm_head. Typical 2026 vocab sizes: GPT-4 cl100k_base around 100K, GPT-4o through GPT-5.5 o200k_base around 200K, Llama 3 and Llama 4 at 128K.

Your answer

**The vocabulary** is the fixed set of token ids a tokenizer can emit. It is a numbered list from 0 to vocab_size minus 1, where each id maps to a specific byte sequence. The model can only ever see and produce ids that are in this list. The size is fixed at tokenizer training time. For BPE, the training algorithm stops merging when the vocabulary reaches the target size, so vocab_size is a hyperparameter you choose before training. Once set, it is baked into the model. Vocabulary size matters because two large pieces of the model depend on it directly. The embedding matrix has one row per token id, each row a vector of size hidden_dim. The lm_head (the output projection) has one column per token id, each column another vector of size hidden_dim. Doubling the vocabulary doubles both of these tables. Typical 2026 sizes. tiktoken's cl100k_base (GPT-4, GPT-3.5-turbo, text-embedding-ada-002) is about 100K tokens. tiktoken's o200k_base (GPT-4o, o1, o3, GPT-5, GPT-5.5) is about 200K tokens. The Llama 3 and Llama 4 tokenizer is 128K tokens. Mistral models use a 32K vocabulary. A bigger vocabulary means each common string can be represented with fewer tokens (lower fertility), but at the cost of larger embedding and output matrices.

**The vocabulary** of a tokenizer is the fixed, numbered set of all token ids it can produce. It is established at tokenizer training time and never changes during model training or inference. For BPE, the vocabulary is the union of the base alphabet (256 byte values for byte-level BPE) and every symbol created by a merge rule, up to the target vocab size. The vocabulary size is a load-bearing hyperparameter. Two model components depend on it directly. The token embedding matrix has shape (vocab_size, hidden_dim): one learned vector per token. The output projection (lm_head) has shape (hidden_dim, vocab_size): every forward pass produces a vocab_size-dimensional logit vector per position. Together these two matrices account for a substantial fraction of the parameter count in modern LLMs, especially smaller ones. Vocab size sits in a tradeoff. Larger vocabularies represent frequent strings with single ids, reducing the number of tokens per word (fertility) and therefore per-token cost. They also reduce sequence length for a given text, which helps attention compute. The cost is parameter count: doubling vocab_size doubles both large vocabulary-aware matrices. Typical 2026 values. OpenAI's tiktoken cl100k_base is approximately 100,256 tokens (GPT-4, GPT-3.5-turbo, text-embedding-ada-002). o200k_base is approximately 200,019 tokens (GPT-4o, o1, o3, GPT-5, GPT-5.5). Meta's Llama 3 and Llama 4 use a 128K-token byte-level BPE vocabulary, deliberately expanded from Llama 2's 32K SentencePiece vocabulary. Mistral retains a 32K vocabulary. The trend is upward as models scale and multilingual coverage matters more.

Quick reference

TL;DR

A tokenizer vocabulary is the fixed numbered list of token ids the tokenizer can emit. Its size is set at training time and directly determines the row count of the embedding matrix and the column count of the lm_head.

Memory aid

Easy to grasp

Imagine the language model only knows a finite list of stamps it can use, like a kid with a sticker book. The sticker book is the vocabulary. Every sticker has a number printed on it, and the model can only ever stick stickers it owns. When the tokenizer reads your prompt, it figures out which stickers to lay out to represent your text. When the model replies, it picks stickers one at a time from the same book. The number of pages in the book is the vocabulary size. A bigger book lets the model represent more things with single stickers (so common words become one sticker instead of three), but every page in the book costs memory inside the model. Picking a vocabulary size is picking how big the sticker book should be before you start training.

Key concepts

Vocabulary Vocab Size Embedding Matrix LM Heado200k_base

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define vocabulary as the numbered list of ids, identify the embedding and lm_head matrices that depend on its size, walk the tradeoff between fertility and parameter count, cite typical 2026 sizes, and distinguish vocabulary from context window and hidden dimension.

Real-world examples

Real products, models, and research that use this idea.

OpenAI's tiktoken cl100k_base has 100,256 tokens (used by GPT-4 and GPT-3.5-turbo); o200k_base has approximately 200,019 tokens (used by GPT-4o, o1, o3, GPT-5, GPT-5.5).
Meta's Llama 3 and Llama 4 use a 128K-token byte-level BPE vocabulary, expanded from Llama 2's 32K SentencePiece vocabulary primarily to improve multilingual and code performance.
Mistral Large 3 retains a 32K-token SentencePiece-based BPE vocabulary, which contributes to higher fertility on non-English text relative to the larger-vocabulary frontier models.
Hugging Face tokenizer.json files explicitly list the full vocabulary; you can inspect any open-weight model's tokens by loading the tokenizer and iterating over its vocab.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf you wanted to add a new special token to a pretrained model, what would actually have to happen?

QWhy did OpenAI move from cl100k_base to o200k_base for GPT-4o and later models?

QWhat are glitch tokens or unreachable tokens, and why do they exist?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Confusing vocabulary size with context-window size. Vocabulary is how many distinct tokens exist; context window is how many tokens fit in one prompt. They are unrelated parameters.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.