There are three classic families of tokenization, and each has a dominant weakness that pushed the field toward the next family.
**Character-level** breaks text into individual characters (or bytes). The vocabulary is tiny (a few hundred entries), and there is never an unknown character. The problem is sequence length. Every word becomes many tokens, so a paragraph that fits in 200 subword tokens might take 1200 character tokens. Attention cost grows with the square of the sequence length, so this is expensive at training and at inference, and it shrinks the effective context window dramatically.
**Word-level** breaks text into whitespace-separated words. Sequences are short, and each token carries clean semantic meaning. The problem is the closed vocabulary. The training corpus contains some finite set of words; anything outside that set becomes a special [UNK] token at inference time, and the information in the original word is lost. New product names, hashtags, typos, and rare technical terms all collide on [UNK].
**Subword** (BPE, WordPiece, Unigram LM) splits text into pieces between characters and whole words. Modern variants are byte-level, so there is never an unknown input. Sequence length is reasonable. This is the Pareto winner that every modern generative LLM uses in 2026.
Byte-level subword still has a quirk worth knowing: a Latin-script bias. Non-Latin characters often encode to multiple bytes in UTF-8, so Chinese, Arabic, or Hindi text uses more tokens per character than English does.
Three tokenization families, three failure modes, and a hybrid family that wins in production.
**Character-level** makes every character (or byte) its own token. The vocabulary is tiny and there is no unknown input. The dominant failure mode is sequence length. The same paragraph that fits in 200 subword tokens may need 1200 character tokens. Attention is O(n^2) in sequence length, so character-level costs several times more at training and inference, and it shrinks the effective context window by the same factor.
**Word-level** splits on whitespace and treats each word as one token. Sequences are short and tokens carry clean meaning. The dominant failure mode is the closed vocabulary. Any word not in the training set becomes [UNK] at inference time and the original information is destroyed. New product names, hashtags, typos, rare jargon, and most morphological inflections collapse onto [UNK].
**Subword** tokenization (BPE, WordPiece, Unigram LM) is the Pareto compromise. Pieces fall between characters and whole words; byte-level variants have no out of vocabulary at all. Sequence length is reasonable (roughly four characters per token under o200k_base). Every modern generative LLM in 2026 uses subword tokenization: GPT-5.5 and Llama 3 and 4 use byte-level BPE, Mistral uses BPE, Gemma uses Unigram LM.
Byte-level subword still has a real failure mode: a Latin-script bias. UTF-8 encodes ASCII in one byte but many non-Latin code points in two, three, or four bytes, and the tokenizer's merges are learned on usually-English-heavy data. Non-Latin scripts (Chinese, Arabic, Hindi, Korean) often need more tokens per character than English, inflating cost and shrinking effective context.
Quick reference
TL;DR
Character-level explodes sequence length. Word-level hits closed vocabulary, so unseen words become [UNK]. Subword is the Pareto winner modern LLMs use; byte-level still has a Latin-script bias.
Imagine three ways to pack a backpack for a trip. The first packer (character) puts each item in its own tiny zip-loc bag. Nothing is missing, but the backpack is now full of bags, and it takes forever to find anything. The second packer (word) only packs items from a fixed packing list. The list does not include the new pair of shoes you just bought, so the shoes get left at home. The third packer (subword) packs in medium chunks: a 'shoe' bag, a 'lace' bag, a 'sole' bag. Anything new can be assembled out of those chunks, and the backpack stays a reasonable size. Modern LLMs use packer three. Packer one is too slow and bulky; packer two cannot handle anything new. Even packer three has a quirk: when working with non-English scripts, each character can take several bytes, so the chunks for those languages end up costing more.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Picking a tokenization family for a new generative LLM in 2026. The family-level question is not live; byte-level subword (BPE or Unigram LM) is the right answer. The interesting choices are within subword: which algorithm, which vocabulary size, byte-level or character-level.
Using character-level tokenization for a long-context model. Sequence length is the dominant cost in attention; character-level multiplies that length and torpedoes the effective context window.
Using word-level tokenization for any open-ended generation task. Generation runs into [UNK] the moment it tries to produce a novel word, which is unacceptable for chat, code, or creative output.
Quick scan · 2–4 min
Everything important, quickly.
4 min: walk through the three classic families, name each dominant failure mode, explain why subword won, note the Latin-script bias that even byte-level subword still has, and place the entire question in the 2026 stack where the family-level choice is no longer live.
Real-world examples
Real products, models, and research that use this idea.
1GPT-5.5 with o200k_base tokenizes about four English characters per token on average, but Chinese text often uses one token per character (or even more), reflecting the Latin-script bias.
2Old character-level models like ByT5 (byte-level T5) exist but are niche; they trade compute for the ability to handle arbitrary bytes without subword vocabulary mismatches.
3Word-level tokenization is essentially extinct in modern generative LLMs; the last serious holdouts were pre-2018 NMT systems, and they all switched to subword once BPE proved its quality.
4tiktoken's o200k_base and cl100k_base are both byte-level BPE; Llama 3 and 4 use byte-level BPE with a 128K vocabulary; this is the production default in 2026.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy exactly does sequence length explode under character-level tokenization, and what does it cost?
Sign in or go Pro to see the approach for this follow-up.
QHow did byte-level BPE solve the out of vocabulary problem completely?
Sign in or go Pro to see the approach for this follow-up.
QIf byte-level subword has a Latin-script bias, how do multilingual models try to mitigate it?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Claiming subword tokenization has no failure mode at all. Byte-level subword has a real, measurable Latin-script bias because non-Latin code points often encode to multiple bytes, inflating token counts for those languages.