Tokenization is how text becomes the integer pieces a model reads, and there are three styles. Word-level uses whole words, character-level uses single characters, and **subword** uses pieces in between.
Word-level has a fixed dictionary built at training time. Any word not in that dictionary, like a new product name, cannot be represented and gets replaced by a single unknown marker, written `[UNK]`. The model loses all information about that word. This is the out of vocabulary, or OOV, problem.
Character-level fixes OOV completely, because you can always spell anything from characters. But it makes sequences very long. A five-letter word becomes five tokens, so a sentence that was ten tokens becomes around fifty. That matters because attention cost grows with the square of sequence length, so long sequences get expensive fast.
Subword is the practical middle ground. Common words and pieces get their own single token, which keeps sequences short, while rare or new words split into known smaller pieces, which avoids OOV. So subword handles anything you throw at it without exploding the length. That is why modern LLMs use it.
Modern LLMs use **subword** tokenization because it resolves the two opposite failures of word-level and character-level approaches at once.
Word-level tokenization builds a closed dictionary at training time. Any word missing from it, a new name or a typo, maps to a single `[UNK]` token and loses all information. That is the out of vocabulary problem.
Character-level tokenization never has OOV, since anything can be spelled from characters, but it explodes sequence length. A five-character word becomes five tokens. Attention cost scales with the square of length, written `O(n^2)`. So a roughly 5x longer sequence means about 25x more attention computation, which is impractical to train.
Subword tokenization takes the middle path. It merges frequent patterns into single tokens, keeping common text short, and splits rare or unseen words into known subpieces, so there is no OOV. Typical English lands near 1.3 tokens per word, close to word-level efficiency, while still representing any string. That Pareto balance, open vocabulary plus tractable length, is why BPE and its variants are the standard.
Quick reference
TL;DR
Subword tokenization is the compromise that handles unseen words by splitting them into known pieces while keeping sequences short, avoiding both word-level OOV failures and character-level length explosion.
Imagine you have a box of building blocks. With word blocks, you can only build things you have a block for, so a brand-new shape you have never seen is impossible to make. With single-dot blocks, you can build anything, but even a tiny shape needs hundreds of dots and takes forever. Subword blocks are medium-sized pieces. Common shapes have their own ready-made block, and anything new you can still build from smaller pieces. That way you can always make the shape, and you do not need a mountain of tiny dots to do it.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A task that is inherently character-level, such as spelling correction or transliteration Subword tokens hide individual characters, so a model has to reconstruct spelling from merged pieces, making fine-grained character manipulation harder.
Serving a low-resource language the tokenizer was barely trained on Subword merges learned from an English-heavy mix fragment that language into many tokens, raising cost and shrinking effective context for those users.
Reliable multi-digit arithmetic where digit grouping must be consistent Subword tokenizers split numbers inconsistently, so the same quantity tokenizes differently across contexts and the model's arithmetic becomes unreliable.
An architecture with sub-quadratic sequence handling and a tiny target vocabulary When length is cheap, character or byte tokenization avoids merge artifacts and gives perfect coverage without the usual quadratic attention penalty.
Real products, models, and research that use this idea.
1OpenAI's GPT-5.5 uses the byte-level BPE tokenizer o200k_base, which keeps English near 1.3 tokens per word while never emitting an unknown token.
2Meta's Llama 4 uses a SentencePiece-derived subword vocabulary so it can represent code, proper nouns, and many languages without an OOV marker.
3Anthropic's Claude models and Google's Gemini 3.1 Pro all rely on subword tokenizers rather than word or character schemes for the same coverage versus length balance.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QByte-level BPE is said to never emit [UNK]. Why is that guaranteed?
Sign in or go Pro to see the approach for this follow-up.
QIf subword is efficient for English at 1.3 tokens per word, why is it unfair to some languages?
Sign in or go Pro to see the approach for this follow-up.
QCould you ever prefer character-level despite the length cost?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Claiming subword always yields shorter sequences than word-level, when in fact word-level is shortest and subword trades a little length to gain open-vocabulary coverage.