Spot the error: 'BPE tokenizers can produce [UNK] when they encounter rare words'

The error is the claim that GPT-4's BPE tokenizer can produce an unknown token `[UNK]` for rare words. It cannot. GPT-2, GPT-3, GPT-4, and Llama use **byte-level BPE**. Their base vocabulary includes all 256 possible byte values. Any text a computer stores is just a sequence of bytes in UTF-8, so every string can be spelled out from those 256 building blocks. When the tokenizer meets a rare or never before seen word, it does not give up. It simply falls back to smaller pieces, possibly down to individual bytes. The result is more tokens, not an unknown token. Where did `[UNK]` come from? Word-level systems like the original word2vec had a fixed word list, so anything off the list became `[UNK]`. WordPiece, used by BERT, can still emit `[UNK]` if every subword split fails. But for byte-level GPT-family tokenizers, `[UNK]` is impossible by design, so telling developers to handle it is misleading.

The flawed claim is that a byte-level BPE tokenizer can emit `[UNK]` on rare words. By construction it cannot. The key is the base vocabulary. Byte-level BPE, used by GPT-2 through GPT-4 and Llama, starts from all **256 byte values**. Every string is UTF-8 bytes, so any input, however exotic, decomposes into tokens drawn from that 256-byte floor. A rare word that lacks a dedicated merge just fragments into shorter pieces or single bytes. The cost is more tokens, never an unknown token. The statement also gets the history backwards. `[UNK]` was a real problem for word-level vocabularies like the original word2vec, where anything off the fixed word list collapsed to unknown. WordPiece, BERT's scheme, can still emit `[UNK]` when no subword split succeeds. Byte-level BPE was designed precisely to eliminate that failure mode. So the advice to 'handle `[UNK]` gracefully' in a GPT-4 pipeline is not just unnecessary, it signals a misunderstanding of byte-level coverage. The real cost of rare input is token inflation, which inflates context usage and price, not unknown tokens.