Both **WordPiece** and BPE build a vocabulary the same way at a high level: start from characters, repeatedly merge an adjacent pair into a new symbol, and stop at a target size. The difference is which pair they pick at each step.
BPE is blunt. It chooses the pair that appears the most times in the corpus. If 'th' shows up more than anything else, 'th' merges first.
WordPiece picks by a likelihood score instead. For a pair of pieces A and B, it computes how often they appear together divided by how often each appears on its own:
$$\text{score}(A,B) = \frac{\text{freq}(AB)}{\text{freq}(A)\,\cdot\,\text{freq}(B)}$$
This rewards pairs that truly belong together, not pairs that are just individually common. A piece like 'e' is everywhere, so any pair containing it has a big denominator and a small score. A genuine fragment like 'qu' has pieces that co-occur far more than chance, so it scores high and merges. The payoff is a vocabulary that lines up with real word structure, which is why BERT uses it.
Both algorithms grow a vocabulary by greedily merging adjacent pairs from a character start, but the **merge criterion** differs. BPE picks the pair with the highest raw count. WordPiece picks the pair that most increases the training corpus likelihood, which reduces to a normalized score:
$$\text{score}(A,B) = \frac{\text{freq}(AB)}{\text{freq}(A)\,\cdot\,\text{freq}(B)}$$
The denominator is the key. It divides out how common each piece is on its own, so the score measures co-dependence, not co-occurrence. A pair like 'e' plus a space is hugely frequent, but its pieces are everywhere, so the ratio stays small. A pair like 's' plus 'h' forming 'sh' co-occurs far above chance, so it scores high.
The practical effect is that WordPiece tends to land on linguistically meaningful subwords, which is why BERT and its descendants use it. The trade-off shows up at encode time: WordPiece does greedy longest-match and can emit `[UNK]` if no vocabulary entry covers a substring, whereas byte-level BPE never does. So WordPiece buys cleaner morphology at the cost of possible unknown tokens.
Quick reference
TL;DR
BPE merges the most frequent pair; WordPiece merges the pair with the highest likelihood gain, freq(AB) / (freq(A)·freq(B)), favoring statistical dependence over raw count.
Imagine you run a sandwich shop and want a one-tap button for the pairs people actually order together. One way is to count every time two items show up in the same order and pick the biggest count. But bread is in almost everything, so 'bread plus anything' wins by sheer volume even when nobody pairs them on purpose.
WordPiece picks differently. It asks: when these two items appear together, is that more often than you'd expect if people chose them independently? Peanut butter and jelly score high because they truly travel as a pair, even if each alone is less common than bread. WordPiece prints buttons for the genuine pairings, not the items that are simply everywhere.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Training a new generative LLM that must ingest arbitrary Unicode, code, and emoji without failures WordPiece can emit unknown tokens on unseen substrings, whereas byte-level BPE guarantees coverage of any input byte.
Multilingual models spanning scripts with sparse training coverage Likelihood-based merges over a thin corpus leave many rare-script substrings without entries, raising the unknown-token rate.
Pipelines standardized on the tiktoken or GPT-style byte-level BPE toolchain Adding a WordPiece tokenizer introduces a second encoding path and continuation-marker conventions that complicate serving and offset mapping.
Quick scan · 2–4 min
Everything important, quickly.
3 min: BPE raw count vs WordPiece likelihood gain + the normalized score + PMI intuition + BERT as the consumer + unknown-token risk vs byte-level BPE.
Real-world examples
Real products, models, and research that use this idea.
1Google's BERT and its successors DistilBERT and ELECTRA tokenize with WordPiece, which is why their vocabularies mark word continuations with a ## prefix.
2The GPT family and Llama 4 use byte-level BPE instead, choosing raw-frequency merges plus guaranteed byte coverage over WordPiece's likelihood criterion.
3HuggingFace tokenizers exposes WordPiece as a first-class model class, so teams fine-tuning BERT-style encoders in 2026 still train it directly.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QCan you derive why maximizing corpus likelihood under a unigram model reduces to ranking pairs by freq(AB) / (freq(A)·freq(B))?
Sign in or go Pro to see the approach for this follow-up.
QWordPiece can emit [UNK]; how would you eliminate that without abandoning the likelihood criterion?
Sign in or go Pro to see the approach for this follow-up.
QHow does WordPiece encoding (greedy longest-match) differ from how its vocabulary was trained?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Calling WordPiece 'BPE on subwords.' Both grow a vocabulary by merging, but BPE merges by raw count while WordPiece merges by a likelihood ratio that divides out how common each piece is.