OOV means an input token the tokenizer cannot represent in its vocabulary; historically the tokenizer emitted a special UNK id. Modern byte-level BPE (GPT-2 through GPT-5.5, Llama 3+, Mistral, DeepSeek V4) initializes its base vocabulary with all 256 byte values, so any UTF-8 input can be decomposed into known tokens with zero loss. There is no UNK by design. OOV is still relevant for legacy WordPiece (BERT-era) and word-level tokenizers.
**OOV** stands for out-of-vocabulary. It refers to an input token that the tokenizer cannot represent in its vocabulary. Historically, when this happened the tokenizer emitted a special UNK token (short for 'unknown'), and the model lost the actual content of that piece of text.
For modern byte-level BPE tokenizers (everything from GPT-2 onward, plus Llama 3+, Mistral, and DeepSeek V4), OOV is essentially a non-issue. The reason is the base alphabet: the vocabulary starts with the 256 possible UTF-8 byte values. Every byte is in the vocabulary by construction, which means any UTF-8 string can be decomposed into known tokens with zero loss. There is no UNK token in tiktoken; there is no UNK token in the Llama 3 tokenizer.
OOV still matters for older tokenizer types. WordPiece (used by BERT and other encoder-only models) does have OOV cases and uses a `[UNK]` token. Pure word-level tokenizers, which were common before sub-word tokenization, had massive OOV issues with any vocabulary they could not enumerate exhaustively.
The related real problem in 2026 is not OOV but fertility. Rare inputs (non-Latin scripts, unusual code, binary data) are representable but require more tokens per character than common inputs. The model can read them; they just cost more.
**OOV (out-of-vocabulary)** describes the case where an input cannot be represented in the tokenizer's vocabulary. Historically, this was a major engineering concern: a word-level or WordPiece tokenizer trained on one corpus would frequently encounter rare or new strings, and the standard fallback was to emit a special UNK token, losing all information about the original content.
Byte-level BPE, introduced by GPT-2 in 2019, eliminates OOV by construction. The base vocabulary contains all 256 possible byte values. Because UTF-8 is a byte encoding and every Unicode character maps to a sequence of bytes, every possible UTF-8 string can be expressed as a sequence of base-level tokens that are guaranteed to be in the vocabulary. The BPE merges add larger sub-word units on top, but the byte-level fallback is always available. No UNK token is needed.
In 2026 this means OOV is not a relevant concept for most production LLM tokenizers. tiktoken (cl100k_base, o200k_base), the Llama 3 and Llama 4 tokenizer, Mistral, and DeepSeek V4 are all byte-level BPE and all lossless. The Anthropic Claude tokenizer is proprietary but documented to be lossless on UTF-8 input.
Where OOV still matters. WordPiece, used by BERT and most encoder-only models from the 2018 to 2022 era, does have OOV cases and uses `[UNK]`. Word-level tokenizers (largely legacy) had severe OOV problems. SentencePiece without the byte-fallback mode also has OOV cases, although the byte_fallback=True configuration (used by Llama 2, T5, and many Hugging Face SentencePiece tokenizers) restores byte-level losslessness.
The modern related concern is fertility, not OOV. Rare scripts and unusual inputs can always be encoded by byte-level BPE, but they use many more tokens per character than common content. This is a cost and capacity issue, not a 'cannot encode' issue.
Quick reference
TL;DR
OOV means an input the tokenizer cannot represent in its vocabulary. For modern byte-level BPE the base vocabulary is the 256 byte values, so any UTF-8 input is representable and there is no UNK token by design.
Imagine a tokenizer is a translator that has to convert your text into a fixed alphabet of stickers. An old word-level translator only had stickers for words it had seen before; if you wrote a brand-new word it had no sticker for, it would slap down a UNK sticker meaning 'I have no idea'. That was the out-of-vocabulary problem, OOV for short. The modern fix is to give the translator a tiny sticker for every possible raw byte value, all 256 of them. Now no matter what you write (a new word, a non-Latin script, an emoji, even random binary data), it can always be broken down into bytes the translator knows. There is no UNK sticker anymore. The downside is that unusual inputs use more stickers to express than common ones, so things like Tamil or Hindi text cost more tokens per word than English. But the model never refuses to encode anything.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Worrying about UNK tokens when working with tiktoken or the Llama 3+ tokenizer. These tokenizers do not have UNK tokens; UNK-handling logic is dead code for them and adds confusion.
Treating high fertility on Tamil or Hindi as an OOV problem. The tokens are perfectly encodable; the issue is they require many tokens per character, which is a cost and capacity problem, not a representability problem.
Comparing OOV behavior across tokenizers as if they all have UNK tokens. OOV behavior varies by tokenizer family. WordPiece has UNK, byte-level BPE does not, SentencePiece depends on configuration. A uniform 'OOV handling' framework does not exist.
Quick scan · 2–4 min
Everything important, quickly.
5 min: define OOV, explain why byte-level BPE has none, contrast the 256-byte base vocabulary with WordPiece's UNK, distinguish OOV from fertility, and name which 2026 tokenizers do and do not have UNK.
Real-world examples
Real products, models, and research that use this idea.
1OpenAI's tiktoken (cl100k_base for GPT-4 and GPT-3.5-turbo, o200k_base for GPT-4o, o1, o3, GPT-5, GPT-5.5) is byte-level BPE with no UNK token; any UTF-8 input encodes losslessly.
2Meta's Llama 3 and Llama 4 tokenizer is tiktoken-style byte-level BPE at 128K vocab; same lossless property, no UNK.
3Hugging Face's BERT WordPiece tokenizer does include `[UNK]` as a real fallback for unknown characters; loading bert-base-uncased and tokenizing rare Unicode shows the UNK appearing.
4Llama 2's SentencePiece tokenizer used byte_fallback=True, so it was lossless despite SentencePiece's potential for OOV in other configurations; Llama 3 moved away from SentencePiece entirely.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf byte-level BPE has no OOV, why do we still talk about tokenization being hard for some languages?
Sign in or go Pro to see the approach for this follow-up.
QWhat does WordPiece do when it encounters an unknown character?
Sign in or go Pro to see the approach for this follow-up.
QWhy did GPT-2 introduce byte-level BPE specifically, and not just expand the character-level vocabulary?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Claiming modern LLMs still suffer from OOV problems. Byte-level BPE eliminates OOV by construction; the related real problem is high fertility on rare scripts, not failure to encode.