Character, word, and subword tokenization each have a dominant failure mode. Match them.

There are three classic families of tokenization, and each has a dominant weakness that pushed the field toward the next family. **Character-level** breaks text into individual characters (or bytes). The vocabulary is tiny (a few hundred entries), and there is never an unknown character. The problem is sequence length. Every word becomes many tokens, so a paragraph that fits in 200 subword tokens might take 1200 character tokens. Attention cost grows with the square of the sequence length, so this is expensive at training and at inference, and it shrinks the effective context window dramatically. **Word-level** breaks text into whitespace-separated words. Sequences are short, and each token carries clean semantic meaning. The problem is the closed vocabulary. The training corpus contains some finite set of words; anything outside that set becomes a special [UNK] token at inference time, and the information in the original word is lost. New product names, hashtags, typos, and rare technical terms all collide on [UNK]. **Subword** (BPE, WordPiece, Unigram LM) splits text into pieces between characters and whole words. Modern variants are byte-level, so there is never an unknown input. Sequence length is reasonable. This is the Pareto winner that every modern generative LLM uses in 2026. Byte-level subword still has a quirk worth knowing: a Latin-script bias. Non-Latin characters often encode to multiple bytes in UTF-8, so Chinese, Arabic, or Hindi text uses more tokens per character than English does.

Three tokenization families, three failure modes, and a hybrid family that wins in production. **Character-level** makes every character (or byte) its own token. The vocabulary is tiny and there is no unknown input. The dominant failure mode is sequence length. The same paragraph that fits in 200 subword tokens may need 1200 character tokens. Attention is O(n^2) in sequence length, so character-level costs several times more at training and inference, and it shrinks the effective context window by the same factor. **Word-level** splits on whitespace and treats each word as one token. Sequences are short and tokens carry clean meaning. The dominant failure mode is the closed vocabulary. Any word not in the training set becomes [UNK] at inference time and the original information is destroyed. New product names, hashtags, typos, rare jargon, and most morphological inflections collapse onto [UNK]. **Subword** tokenization (BPE, WordPiece, Unigram LM) is the Pareto compromise. Pieces fall between characters and whole words; byte-level variants have no out of vocabulary at all. Sequence length is reasonable (roughly four characters per token under o200k_base). Every modern generative LLM in 2026 uses subword tokenization: GPT-5.5 and Llama 3 and 4 use byte-level BPE, Mistral uses BPE, Gemma uses Unigram LM. Byte-level subword still has a real failure mode: a Latin-script bias. UTF-8 encodes ASCII in one byte but many non-Latin code points in two, three, or four bytes, and the tokenizer's merges are learned on usually-English-heavy data. Non-Latin scripts (Chinese, Arabic, Hindi, Korean) often need more tokens per character than English, inflating cost and shrinking effective context.