How does byte-level BPE guarantee no unknown tokens, and what is the cost of that guarantee?

Older word-based tokenizers had a famous weakness: if they met a word they never saw in training, they had to emit a special [UNK] (unknown) token and lose the information. This is the out-of-vocabulary, or OOV, problem. **Byte-level BPE** dodges this entirely. Before learning any merges, it seeds its vocabulary with all 256 possible byte values, the numbers 0 through 255. Every piece of text on a computer is stored as UTF-8 bytes, and every byte is in that set, so any input can always be spelled out byte by byte. There is simply nothing left to be 'unknown'. The cost shows up for rare scripts. BPE learns its shortcut merges from training data, which was mostly English. A script like Arabic or Chinese rarely earns those shortcuts, so its characters fall all the way back to individual bytes. A short word can become a long string of tokens, eating more of the context window than the same-length English word.

**Byte-level BPE** guarantees zero unknown tokens by construction: it seeds the base vocabulary with all 256 byte values before learning a single merge. The argument is airtight. Any string, in any language, is stored as UTF-8 bytes, and every UTF-8 byte is one of those 256. So in the worst case the tokenizer spells the input out byte by byte. There is no input it cannot represent, which means it never needs an [UNK] token. The out-of-vocabulary problem that plagued word-level and WordPiece tokenizers disappears. The guarantee is not free. BPE learns its merges from frequency in training data, and that data skewed heavily English. Common English patterns merged into compact tokens; rare scripts like Arabic or Chinese earned few or no merges, so their characters fall back toward individual bytes. The trade-off is therefore representational completeness for sequence length. Non-Latin text becomes long token sequences, which inflates cost and consumes the context window faster. You never lose information, but you pay for unfamiliar input in tokens rather than in [UNK] symbols.