BPE vs WordPiece vs SentencePiece
Three subword tokenizers powering modern LLMs
BPE merges frequent pairs, WordPiece picks merges by likelihood, SentencePiece is a training framework that treats input as raw bytes (or unicode) and can produce BPE or unigram tokenizers.
BPE
Glossary →Byte-Pair Encoding starts from characters and iteratively merges the most frequent adjacent pair. GPT-style models use it (often as byte-level BPE).
Best for: Simple, frequency-driven vocabularies.
WordPiece
Glossary →Similar to BPE, but instead of picking the most frequent pair to merge, WordPiece picks the pair that most improves training data likelihood. Used by BERT.
Best for: Likelihood-optimal subword vocabularies.
SentencePiece
Glossary →A tokenizer training framework that operates on raw text (whitespace becomes a normal character), supporting both BPE and unigram-LM training. Language-agnostic. Used by T5, Llama, and many multilingual models.
Best for: Language-agnostic pipelines and byte-level input.
At a glance
| Dimension | BPE | WordPiece | SentencePiece |
|---|---|---|---|
| Merge criterion | Pair frequency | Likelihood gain | BPE or unigram |
| Reversible detokenization | Depends on preprocessing | Depends on preprocessing | Yes (whitespace as character) |
| Prefix convention | ▁ (byte-BPE) | ## for continuations | ▁ for word starts |
| Language coverage | Language-agnostic (byte-level) | Typically pre-tokenized languages | Language-agnostic |
| Used by | GPT-2/3/4, Llama byte-BPE | BERT, DistilBERT | T5, mT5, Llama |
| Best for | Simple frequency-driven vocab | Maximum likelihood on training corpus | Reversibility + multi-language |
Key differences
- 1BPE picks merges by frequency; WordPiece by likelihood
- 2SentencePiece is a framework, not a fourth algorithm; it can train BPE or unigram
- 3SentencePiece treats whitespace as a character, enabling reversible detokenization
- 4Byte-level BPE (GPT-2, Llama) removes the need for a preprocessing step by working over raw bytes
- 5WordPiece uses ## prefix for continuations; BPE/SentencePiece typically use ▁ for word starts
In the interview
- Claiming SentencePiece is an algorithm separate from BPE/unigram
- Not knowing byte-level BPE removes the need for pre-tokenization
- Missing that WordPiece's ## prefix marks continuations, not word starts
How to choose
New model → SentencePiece (BPE or unigram). Matching an existing checkpoint → whatever it used.
Common misconceptions
Myth: SentencePiece is a fourth tokenizer algorithm.
Reality: It's a training framework that can build BPE or unigram tokenizers. What makes it distinctive is its reversibility on raw text.
Myth: BPE and WordPiece are effectively identical.
Reality: They differ on merge criterion (frequency vs likelihood) and on prefix conventions, which matters when you're loading pretrained tokenizers.
Memory aid
BPE glues the loudest pair. WordPiece glues the pair that most improves the story. SentencePiece is the workshop where you build either glue.
Can you combine them?
Not typically; you pick one tokenizer per model. You can use SentencePiece to train a BPE tokenizer, effectively combining the framework and the algorithm.