Why does BPE tokenization use subwords instead of words or characters?
Subword tokenization keeps vocab compact, handles unseen words by splitting them into known pieces, and keeps sequences short enough for attention to stay cheap.
Imagine teaching a robot to read English. One option is to give it a dictionary with every word in it. That dictionary would be huge, and the moment someone types a brand new slang word, the robot is stuck. Another option is to teach the robot one letter at a time. Now the robot knows everything, but a single sentence becomes hundreds of pieces and the robot gets tired before it reaches the end. Subword tokenization, which is what BPE does, splits text into common chunks like 'token', 'ization', and 'un'. Common words stay as one chunk, weird new words get broken into pieces the robot already knows, and the input stays short enough to think about.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Every LLM you have ever used starts with the same first step: chop text into a sequence of integer IDs the embedding layer can look up. The choice of how to chop is one of the few decisions baked into the model that you cannot change after pre-training. Pick wrong and you either pay forever in vocabulary size or forever in sequence length.
This question asks why subword tokenization, and specifically Byte Pair Encoding, became the universal default. The answer is not that BPE is clever (it is a remarkably simple greedy algorithm). The answer is that the two natural alternatives, word level and character level, each fail in ways that compose badly with transformer attention.
What word-level tokenization buys you, and what it costs
A word-level tokenizer maps each whitespace-separated word to a unique ID. The vocabulary is your dictionary. This sounds clean until you count: English has roughly 200k common words before you include plurals, possessives, typos, proper nouns, code identifiers, URLs, and emoji. Once you support multiple languages or any user-generated content, the long tail becomes infinite.
The production answer historically was to cap the vocabulary at 30k or 50k and route everything else to a single UNK token. This silently destroys information. The model sees UNK for every novel word, including the brand names, product SKUs, and slang that often carry the actual signal of a sentence.
The deeper problem. Even within the cap, embedding tables grow linearly with vocabulary. A 50k word vocabulary with d_model = 4096 is already 200 million parameters, which is fine. But you have not solved the OOV problem, only hidden it.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import tiktoken
# cl100k_base is the BPE vocab used by GPT-4 family models
enc = tiktoken.get_encoding("cl100k_base")
text = "Tokenization is the first layer of every LLM."
ids = enc.encode(text)
print(len(text), "chars ->", len(ids), "tokens")
print([enc.decode([i]) for i in ids])
# Common words stay whole, rare patterns split.
# 'Tokenization' is one token; an unseen word like 'flibbertigibbet'
# would split into several known subwords.Real products, models, and research that use this idea.
- OpenAI's tiktoken powers the BPE tokenizer behind GPT-5.5 and the o-series, with cl100k_base and o200k_base vocabularies.
- Google's SentencePiece is the tokenizer used by Gemini, T5, and many multilingual checkpoints because it handles whitespace as a regular symbol.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you choose vocab_size for a new LLM you are training from scratch?
Tie it to compute budget and training corpus mix. Larger vocab shortens sequences but inflates the embedding table and output softmax cost.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating BPE as a compression trick rather than the contract that determines how every downstream layer perceives the input text.
60 second bullets to scan on the way to the call.
Why word-level tokenization fails on open-domain text
Why character-level tokenization fails on attention compute
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.