BPE and WordPiece both merge subwords. What is the one thing they disagree on?

MCQ·Easy·4.0 · 0·~1 min·Asked atIBM Lyzr·Relevant atGoogle

Attempt it

**BPE and WordPiece** are both subword (smaller than whole words) tokenizer training algorithms. They both start from a base alphabet (bytes or characters). They both grow a vocabulary by repeatedly picking two adjacent pieces and merging them into a new piece. They differ in one place: how they pick which pair to merge next. BPE picks the pair that appears most often in the training corpus. Count every adjacent pair across the corpus, pick the pair with the highest count, merge it, repeat. That is the whole algorithm. WordPiece picks the pair whose merge would most increase the likelihood of the corpus under a unigram language model (a model that scores pieces independently). In practice this works out to maximizing the score freq(AB) divided by freq(A) times freq(B). A pair that occurs often, but where each piece is rare on its own, scores high. A pair where both pieces are already very common scores low, even if the pair itself is frequent. BPE is what generative LLMs use: GPT-4 and GPT-4o (tiktoken cl100k_base and o200k_base), Llama 3 and 4, Mistral. WordPiece is the BERT-family choice and shows up mostly in research and legacy encoder pipelines.

**BPE and WordPiece** are both iterative subword tokenizer training algorithms. They share the same skeleton. Start from a base alphabet and count adjacent pairs across the corpus. Pick a winning pair, merge it everywhere, and repeat until the vocabulary reaches the target size. The headline difference is the scoring function used to pick the winning pair. BPE picks the pair with the highest raw frequency. This is a pure count-based criterion, originally borrowed from data compression (Gage 1994, Sennrich et al. 2016). The selected pair is whichever adjacency shows up most often in the current corpus state. WordPiece picks the pair whose merge most increases corpus likelihood under a unigram language model. The full derivation reduces to picking the pair (A, B) that maximizes freq(AB) divided by freq(A) times freq(B). This ratio is high when AB occurs together far more often than chance would predict given how often A and B occur separately. WordPiece therefore prefers merges that are statistically surprising, not merely common. In production, BPE dominates generative LLMs. GPT-4o and GPT-5 use tiktoken with the o200k_base BPE encoder. Llama 3 uses a 128K BPE vocabulary, and Mistral 7B uses a SentencePiece BPE. WordPiece is the BERT-family algorithm and is now mostly research or legacy encoder tooling. Both encode greedily: BPE replays its merge list, and WordPiece takes the longest matching vocabulary piece first.

TL;DR

BPE picks the next merge by raw pair frequency; WordPiece picks the merge that maximally increases corpus likelihood, scored as freq(AB) over freq(A) times freq(B).

Memory aid

Easy to grasp

Picture two kids stacking blocks. Both kids start with a pile of letter blocks and a rule: each turn, glue two blocks together to make a bigger block, and add the new shape to your toolbox. The first kid (BPE) glues whichever two blocks happen to sit next to each other most often. That is simple counting. The second kid (WordPiece) is fussier. It only glues two blocks together if the pair shows up much more often than you would expect from how often each block shows up on its own. That ratio rewards merges that are surprising. After many turns, both kids have a toolbox of bigger blocks they can use to build any word. They tend to pick similar blocks for common words but diverge on rarer combinations. That is why a text tokenized with BPE and the same text tokenized with WordPiece end up looking subtly different.

Key concepts

Byte Pair Encoding WordPiece Subword Tokenization Merge Rule

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

BPE and WordPiece are the two best-known subword tokenizer training algorithms. They look almost identical from a distance: both grow a vocabulary by iteratively picking adjacent pairs and merging them into a new token. The distinction is small but important. It is now mostly a historical landmark that explains why generative LLMs and BERT-family models tokenize text differently.

The shared training skeleton

Both algorithms start from a base alphabet. For classical BPE and WordPiece this is the set of Unicode characters in the corpus. For byte-level BPE (the GPT-2 variant most generative LLMs now use) it is the 256 possible byte values.

From that base, both algorithms run an iterative loop. On each iteration, they look at every adjacent pair of pieces across the entire training corpus and count or score those pairs. They pick a winning pair and merge every occurrence of that pair into a new piece. That piece gets added to the vocabulary. Then they recount and pick again. The loop continues until the vocabulary reaches the target size, which is usually a hyperparameter chosen up front (32K, 50K, 128K, 200K, and so on).

BPE saves a learned, ordered list of merges and reapplies that list greedily to any input string at encoding time. WordPiece saves only the final vocabulary. It encodes each word by taking the longest matching vocabulary piece first, then repeating on the rest. Both algorithms are deterministic at encoding time and produce one segmentation per input. The headline split between them is how the winning pair is chosen during training.

BPE: the frequency criterion

WordPiece: the likelihood criterion

Why the difference matters and why it mostly does not

Where each is used in production

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

GPT-4o, GPT-5, and o3 use BPE via tiktoken's o200k_base encoding with a 200K vocabulary; every tiktoken encoding is BPE.
Llama 3 uses BPE with a 128K vocabulary (Llama 4 keeps BPE with about 200K), having switched away from the SentencePiece tokenizer used in Llama 2.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhere does the freq(AB) over freq(A) times freq(B) formula come from?

Start from the unigram language model likelihood of the corpus. Write the change in log-likelihood from merging pair (A, B) into a new token AB. Then show that maximizing that delta reduces to picking the pair with the largest score freq(AB) divided by freq(A) times freq(B). The ratio is the closed-form solution under the unigram independence assumption.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Saying the main difference is how they segment at inference time. The headline difference is how they pick merges during training. Encoding differs too: BPE replays its merge list, while WordPiece takes the longest matching vocabulary piece.

60 second bullets to scan on the way to the call.

State the shared skeleton: iterative pair selection, merge, repeat to a target vocabulary size.
Name BPE's merge criterion in one phrase.

Primary sources. Browse if you want the original framing.