Explain why word-level and character-level tokenization both fail for modern LLMs, and how subword tokenization resolves each failure mode.

There are three ways to split text into tokens, and two of them fail badly for large models. Word-level tokenization uses a fixed dictionary built at training time. Any word not in that dictionary, like a new name or a technical term, becomes a single unknown token written `[UNK]`. The model loses everything about that word. This is the out of vocabulary problem, and it happens constantly because new words appear all the time. Character-level tokenization never has that problem, because anything can be spelled from characters. The catch is length. A ten-word sentence at about five characters per word becomes roughly fifty tokens, and with spaces closer to sixty. That is about a six-times expansion over word-level. Why does length hurt so much? Attention compares every token with every other token, so its cost grows with the square of the sequence length. A six-times longer sequence costs around thirty-six times more attention compute. **Subword** tokenization is the fix. It gives common words and pieces their own single token, keeping sequences short, and splits rare or new words into known smaller pieces, so there is no `[UNK]`. English text averages about 1.3 tokens per word, close to word-level, while still handling anything. That is why it is the standard.

Word-level and character-level fail in opposite directions, and **subword** is the point between them. Word-level builds a closed vocabulary at training time. Any word missing from it maps to a single `[UNK]` token and loses all information, so proper nouns, new terms, and typos vanish at the input. That is the out of vocabulary failure. Character-level never has OOV, since anything can be spelled from characters, but it explodes length. A 10-word sentence at about 5 characters per word is roughly 50 tokens, near 60 with spaces, versus about 10 for word-level, a 5-to-6x expansion. That matters because attention scales quadratically: $$\text{attention cost} \sim O(n^2)$$ So a 6x longer sequence is about 36x more attention compute and memory, which is impractical to train. Subword tokenization, via BPE, WordPiece, or SentencePiece, merges frequent patterns into single tokens to keep sequences short, and splits rare or unseen words into known subpieces to stay open-vocabulary. English averages about 1.3 tokens per word, close to word-level efficiency, while representing any string. Open vocabulary plus tractable length is exactly the combination word-level and character-level each give up half of.