A language model team is debating whether to use word-level, character-level, or subword tokenization. Trace the specific failure modes of word-level and character-level approaches, quantifying the character-level sequence-length penalty on a typical 10-word English sentence, and explain why subword is the standard choice.
Free · 2 AI evals / day
There are three ways to split text into tokens, and two of them fail badly for large models.
Word-level tokenization uses a fixed dictionary built at training time. Any word not in that dictionary, like a new name or a technical term, becomes a single unknown token written `[UNK]`. The model loses everything about that word. This is the out of vocabulary problem, and it happens constantly because new words appear all the time.
Character-level tokenization never has that problem, because anything can be spelled from characters. The catch is length. A ten-word sentence at about five characters per word becomes roughly fifty tokens, and with spaces closer to sixty. That is about a six-times expansion over word-level.
Why does length hurt so much? Attention compares every token with every other token, so its cost grows with the square of the sequence length. A six-times longer sequence costs around thirty-six times more attention compute.
**Subword** tokenization is the fix. It gives common words and pieces their own single token, keeping sequences short, and splits rare or new words into known smaller pieces, so there is no `[UNK]`. English text averages about 1.3 tokens per word, close to word-level, while still handling anything. That is why it is the standard.
Word-level and character-level fail in opposite directions, and **subword** is the point between them.
Word-level builds a closed vocabulary at training time. Any word missing from it maps to a single `[UNK]` token and loses all information, so proper nouns, new terms, and typos vanish at the input. That is the out of vocabulary failure.
Character-level never has OOV, since anything can be spelled from characters, but it explodes length. A 10-word sentence at about 5 characters per word is roughly 50 tokens, near 60 with spaces, versus about 10 for word-level, a 5-to-6x expansion. That matters because attention scales quadratically:
$$\text{attention cost} \sim O(n^2)$$
So a 6x longer sequence is about 36x more attention compute and memory, which is impractical to train.
Subword tokenization, via BPE, WordPiece, or SentencePiece, merges frequent patterns into single tokens to keep sequences short, and splits rare or unseen words into known subpieces to stay open-vocabulary. English averages about 1.3 tokens per word, close to word-level efficiency, while representing any string. Open vocabulary plus tractable length is exactly the combination word-level and character-level each give up half of.
Quick reference
TL;DR
Word-level loses unseen words to [UNK]; character-level explodes length into quadratic attention cost; subword splits rare words into known pieces and merges common ones, staying near word-level length.
Think of packing a suitcase. Whole-outfit packing is fast, but if you need an outfit you never packed, you simply cannot wear it. Packing one thread at a time means you can make any outfit, but the suitcase needs thousands of threads and weighs a ton. Subword packing uses pre-made sleeves, collars, and panels. Common outfits snap together from a few pieces, and anything new you can still assemble from the panels. You can always dress for the occasion, and the suitcase stays light enough to carry.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Character-level tasks like spelling correction, transliteration, or one letter at a time generation Subword merges hide individual characters, so the model must reconstruct spelling from chunks, which makes fine-grained character work error-prone.
Deploying an English-trained subword tokenizer for a low-resource target language The learned merges fragment that language heavily, so its fertility climbs and users pay more per word and lose effective context, eroding the efficiency claim.
Numeric-heavy workloads needing consistent digit grouping for arithmetic Subword tokenizers split numbers inconsistently, so the same value tokenizes differently across contexts and the model's arithmetic reliability drops.
Architectures with sub-quadratic sequence handling and tiny vocabularies When length is cheap, byte or character tokenization gives perfect coverage and avoids merge artifacts without paying the quadratic attention penalty subword exists to avoid.
Real products, models, and research that use this idea.
1OpenAI's GPT-5.5 tokenizer o200k_base is byte-level BPE, keeping English near 1.3 tokens per word with no unknown token across arbitrary Unicode.
2Meta's Llama 4 uses a SentencePiece-based subword vocabulary so it handles code, names, and many languages without an OOV marker.
3Anthropic's Claude and Google's Gemini 3.1 Pro both use subword tokenizers rather than word or character schemes to balance coverage against sequence length.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QYou computed about 36x more attention cost for character-level. Does anything reduce that in modern architectures?
Sign in or go Pro to see the approach for this follow-up.
QSubword averages 1.3 tokens per word for English. Why can a different language see 4 or 5?
Sign in or go Pro to see the approach for this follow-up.
QIf character-level is so costly, why do some production systems still tokenize at the byte level?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Naming the character-level length penalty but forgetting it multiplies attention cost quadratically, so a 5x longer sequence is roughly 25x more compute, not 5x.