Why does BPE tokenization use subwords instead of words or characters?

Tokenization is how raw text becomes the integer IDs the model actually reads. The choice of granularity matters more than it looks. Word-level tokenizers give each whole word an ID. English has hundreds of thousands of distinct words once you count plurals, typos, names, and code identifiers. You either ship a giant embedding table or you mark anything unseen as `UNK`, which silently destroys information. New slang, product names, and email addresses all collapse to the same hole. Character-level tokenizers go the other way. The vocabulary shrinks to a few hundred symbols and nothing is ever out-of-vocabulary. The cost shows up in sequence length: a 50-word sentence becomes around 250 characters, and transformer attention is `O(n^2)` in sequence length. You pay roughly 25 times more compute for the same paragraph. **Byte Pair Encoding** sits between these extremes. It learns a vocabulary of frequent character sequences from a corpus, so common words stay whole, rare words split into a handful of known pieces, and nothing ever falls off the edge.

**Byte Pair Encoding** sits between word-level and character-level tokenization to dodge the failure mode of each. Word tokenizers need an enormous vocabulary and still produce `UNK` tokens on anything they have not seen. Character tokenizers handle everything but blow up sequence length, which is fatal because attention compute scales as `O(n^2)`. BPE learns a fixed-size vocabulary (commonly around 32k to 100k entries) by repeatedly merging the most frequent adjacent pair of symbols in a training corpus. Common words like `the`, `language`, and ` model` become single tokens. Rare words like `tokenization` decompose into a few familiar pieces such as `token` and `ization`. Anything truly novel still encodes byte by byte, so the model never sees an out-of-vocabulary symbol. The practical result for a 2026 LLM is that English compresses to roughly four characters per token, code compresses well, and rare proper nouns degrade gracefully instead of vanishing. That is why GPT, Claude, Llama, and Mistral all ship a subword tokenizer rather than a word-level or character-level one.