Oov

Older tokenizers had an OOV problem. Why did byte-level BPE kill it?

Character, word, and subword tokenization each have a dominant failure mode. Match them.

Why do modern LLMs use subword tokenization instead of word-level or character-level approaches?

Spot the error: 'BPE tokenizers can produce [UNK] when they encounter rare words'

Predict how many tokens the rainbow flag emoji 🏳️‍🌈 uses in a byte level BPE tokenizer with no emoji in training data.

How does byte-level BPE guarantee no unknown tokens, and what is the cost of that guarantee?