OOV means an input the tokenizer cannot represent in its vocabulary. For byte-level BPE the base vocabulary is the 256 byte values, so any UTF-8 input is representable and there is no UNK token by design.
Imagine a tokenizer is a translator that has to convert your text into a fixed alphabet of stickers. An old word-level translator only had stickers for words it had seen before. If you wrote a new word, it put down a UNK sticker that means 'I have no idea'. That was the out-of-vocabulary problem, OOV for short. The modern fix is to give the translator a tiny sticker for every possible raw byte value, all 256 of them. Now no matter what you write (a new word, a non-Latin script, an emoji, even random binary data), it can always be broken down into bytes the translator knows. There is no UNK sticker anymore. The downside is that unusual inputs use more stickers to express than common ones. So things like Tamil or Hindi text cost more tokens per word than English. But the model never refuses to encode anything.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
OOV (out-of-vocabulary) used to be a major engineering concern for NLP systems. Pre-2019 tokenizers often met text they could not represent. The standard answer was an UNK token that quietly destroyed information. Byte-level BPE, introduced by GPT-2, structurally removed the problem. OOV is now largely a historical artifact for production LLM tokenizers. The concept still matters, because legacy systems and confused mental models both persist.
OOV, defined and historically situated
OOV stands for out-of-vocabulary. It refers to the case where an input string contains a unit the tokenizer's vocabulary cannot represent. The classic response is to emit a special UNK token (variously spelled [UNK], <unk>, or similar) as a placeholder for the unrepresentable content. The model then sees UNK in place of the actual text, losing all information about what was there.
OOV was a major issue in pre-subword NLP. Word-level tokenizers (a list of known words plus UNK) would emit UNK for every unknown input. That included a new word, a misspelling, a proper noun the corpus had not contained, or a word in a different language. Coverage was a constant struggle: expanding the vocabulary to cover more words made the embedding table balloon, but every cut was a real information loss.
WordPiece (first built for Japanese and Korean voice search in 2012, later made popular by BERT in 2018) reduced the problem by using sub-word units. Frequent words got their own ids; rare words were decomposed into shorter pieces (often morpheme-like fragments and the special ## continuation prefix). This pushed the OOV boundary down to the character level. WordPiece only emits UNK when a character itself is unknown, which is much rarer than unknown words but not impossible.
Byte Pair Encoding (BPE) at the character level had similar trade-offs: it reduced OOV but did not remove it. The character-level base alphabet was bounded by what the training corpus contained. Characters not in the training corpus would be UNK at inference time.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI's tiktoken (cl100k_base for GPT-4 and GPT-3.5-turbo, o200k_base for GPT-4o, o1, o3, GPT-5) is byte-level BPE with no UNK token; any UTF-8 input encodes losslessly.
- Meta's Llama 3 tokenizer is tiktoken-style byte-level BPE at 128K vocab (Llama 4 uses about 200K). It has the same lossless property and no UNK.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf byte-level BPE has no OOV, why do we still talk about tokenization being hard for some languages?
The hard part shifted from representability to fertility. Tamil and similar scripts encode to far more tokens per character than English under cl100k_base. The model can read them but uses more of the context window and costs more per character. Walk through how a 200-character Tamil sentence might tokenize to about 270 tokens versus around 45 tokens for the same-length English sentence.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming modern LLMs still suffer from OOV problems. Byte-level BPE removes OOV by construction; the related real problem is high fertility on rare scripts, not failure to encode.
60 second bullets to scan on the way to the call.
Define OOV in one sentence.
Explain why byte-level BPE removes OOV by construction.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.