A tokenizer is 'byte level'. What does that buy you over a character level one?

A **byte-level** tokenizer buys you three concrete advantages over a character-level one, and understanding them explains why byte-level became the dominant LLM design. First, guaranteed coverage with no out of vocabulary case. The base alphabet is the 256 possible byte values (0x00 through 0xFF). Every Unicode character encodes to one to four bytes in UTF-8, and every one of those bytes is in the vocabulary. Tamil, emoji, code, even raw binary: everything encodes. A character-level tokenizer's alphabet is bounded by training, so unseen characters hit the UNK token. Second, a fixed base alphabet regardless of script. Character-level tokenizers need an alphabet that grows with every new script. Byte-level always starts with exactly 256 entries. The BPE merge algorithm builds longer tokens on top, but the foundation never changes. Third, lossless round tripping. Encode to token ids, decode back, and you get the original bytes exactly. No normalization loss, no information dropped. The tradeoff is uneven fertility across scripts. English text gets short merged tokens (often whole words); rare scripts rely on single-byte fragments and pay a 3 to 5x per-character cost. This is the tokenizer tax. It narrows with larger vocabularies (o200k_base, Llama 3 at 128K) but has not disappeared.

**Byte-level** describes a tokenizer whose base alphabet is the 256 possible UTF-8 byte values, rather than the set of Unicode characters in the training corpus. Every byte is in the base vocabulary by construction; the BPE merge algorithm adds longer sub-word tokens on top. The defining property is losslessness on UTF-8 input. UTF-8 encodes every Unicode character as one to four bytes; every byte is a token id in the base vocabulary. Therefore any UTF-8 string encodes as a sequence of byte-level tokens with no OOV case and no information loss. This extends to arbitrary binary data, even though the model has not been trained to understand it. Byte-level was introduced by GPT-2 in 2019 and has become the dominant family for LLM tokenization. In 2026: tiktoken (cl100k_base, o200k_base), Llama 3 and Llama 4 (128K vocab), Mistral, and DeepSeek V4 all use byte-level BPE. Claude's tokenizer is proprietary but documented as lossless on UTF-8. Not all sub-word tokenizers are byte-level. SentencePiece in default configuration is character-level with OOV. With character_coverage=1.0 and byte_fallback=True, SentencePiece achieves byte-level losslessness via fallback; Llama 2 used this. WordPiece (BERT family) is character-level with a real `[UNK]` token. The tradeoff is uneven fertility. A Chinese character takes three UTF-8 bytes and may consume multiple tokens, while a common English word can be one token. Larger vocabularies reduce the gap but do not eliminate it.