Why do modern LLMs use subword tokenization instead of word level or character level approaches?

Tokenization is how text becomes the integer pieces a model reads, and there are three styles. Word-level uses whole words, character-level uses single characters, and **subword** uses pieces in between. Word-level has a fixed dictionary built at training time. Any word not in that dictionary, like a new product name, cannot be represented and gets replaced by a single unknown marker, written `[UNK]`. The model loses all information about that word. This is the out of vocabulary, or OOV, problem. Character-level fixes OOV completely, because you can always spell anything from characters. But it makes sequences very long. A five-letter word becomes five tokens, so a sentence that was ten tokens becomes around fifty. That matters because attention cost grows with the square of sequence length, so long sequences get expensive fast. Subword is the practical middle ground. Common words and pieces get their own single token, which keeps sequences short, while rare or new words split into known smaller pieces, which avoids OOV. So subword handles anything you throw at it without exploding the length. That is why modern LLMs use it.

Modern LLMs use **subword** tokenization because it resolves the two opposite failures of word-level and character-level approaches at once. Word-level tokenization builds a closed dictionary at training time. Any word missing from it, a new name or a typo, maps to a single `[UNK]` token and loses all information. That is the out of vocabulary problem. Character-level tokenization never has OOV, since anything can be spelled from characters, but it explodes sequence length. A five-character word becomes five tokens. Attention cost scales with the square of length, written `O(n^2)`. So a roughly 5x longer sequence means about 25x more attention computation, which is impractical to train. Subword tokenization takes the middle path. It merges frequent patterns into single tokens, keeping common text short, and splits rare or unseen words into known subpieces, so there is no OOV. Typical English lands near 1.3 tokens per word, close to word-level efficiency, while still representing any string. That Pareto balance, open vocabulary plus tractable length, is why BPE and its variants are the standard.