BPE and WordPiece both merge subwords. What is the one thing they disagree on?

MCQ·Easy·4.0 · 0·~1 min·Asked atIBM Lyzr·Relevant atGoogle

Attempt it

**BPE and WordPiece** are both subword tokenizer training algorithms. They both start from a base alphabet (bytes or characters), and they both grow a vocabulary by repeatedly picking two adjacent pieces and merging them into a new piece. They differ in one place: how they pick which pair to merge next. BPE picks the pair that appears most often in the training corpus. Count every adjacent pair across the corpus, pick the pair with the highest count, merge it, repeat. That is the whole algorithm. WordPiece picks the pair whose merge would most increase the likelihood of the corpus under a unigram language model. In practice this works out to maximizing the score freq(AB) divided by freq(A) times freq(B). A pair that occurs often, but where each piece is rare on its own, scores high. A pair where both pieces are already very common scores low, even if the pair itself is frequent. In 2026, BPE is what generative LLMs use: GPT-5.5 (tiktoken cl100k_base and o200k_base), Llama 3 and 4, Mistral. WordPiece is the BERT-family choice and shows up mostly in research and legacy encoder pipelines.

**BPE and WordPiece** are both iterative subword tokenizer training algorithms. They share the same skeleton: start from a base alphabet, count adjacent pairs across the corpus, pick a winning pair, merge it everywhere, repeat until the vocabulary reaches the target size. The headline difference is the scoring function used to pick the winning pair. BPE picks the pair with the highest raw frequency. This is a pure count-based criterion, originally borrowed from data compression (Gage 1994, Sennrich et al. 2016). The selected pair is whichever adjacency shows up most often in the current corpus state. WordPiece picks the pair whose merge most increases corpus likelihood under a unigram language model. The full derivation reduces to picking the pair (A, B) that maximizes freq(AB) divided by freq(A) times freq(B). This ratio is high when AB occurs together far more often than chance would predict given how often A and B occur separately. WordPiece therefore prefers merges that are statistically surprising, not merely common. In 2026 production, BPE dominates generative LLMs: GPT-5.5 uses tiktoken with the o200k_base BPE encoder, Llama 3 and 4 use a 128K BPE vocabulary, Mistral uses a SentencePiece BPE. WordPiece is the BERT-family algorithm and is now mostly research or legacy encoder tooling. Both apply their learned merges greedily at encoding time; the algorithmic split is purely about training.

TL;DR

BPE picks the next merge by raw pair frequency; WordPiece picks the merge that maximally increases corpus likelihood, scored as freq(AB) over freq(A) times freq(B).

Memory aid

Easy to grasp

Picture a kid stacking blocks. Both kids start with a pile of letter blocks and a rule: each turn, glue two blocks together to make a bigger block, and add the new shape to your toolbox. The first kid (BPE) glues whichever two blocks happen to sit next to each other most often. Simple counting. The second kid (WordPiece) is fussier. It only glues two blocks together if the pair shows up much more often than you would expect from how often each block shows up on its own. That ratio rewards merges that are genuinely surprising. After many turns, both kids have a toolbox of bigger blocks they can use to build any word. They tend to pick similar blocks for common words but diverge on rarer combinations, which is why a text tokenized with BPE and the same text tokenized with WordPiece end up looking subtly different.

Key concepts

Byte Pair Encoding WordPiece Subword Tokenization Merge Rule

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

BPE and WordPiece are the two best-known subword tokenizer training algorithms. They look almost identical from a distance: both grow a vocabulary by iteratively picking adjacent pairs and merging them into a new token. The distinction is small but consequential, and in 2026 it is mostly a historical landmark that explains why generative LLMs and BERT-family models tokenize text differently.

This deep dive unpacks the shared skeleton, names the precise difference in the merge-selection rule, derives where the WordPiece ratio comes from, and places both algorithms in the modern production landscape.

The shared training skeleton

Both algorithms start from a base alphabet. For classical BPE and WordPiece this is the set of Unicode characters in the corpus. For byte-level BPE (the GPT-2 variant most generative LLMs now use) it is the 256 possible byte values.

From that base, both algorithms run an iterative loop. On each iteration, they look at every adjacent pair of pieces across the entire training corpus, count or score those pairs, pick a winning pair, and merge every occurrence of that pair into a new piece that gets added to the vocabulary. Then they recount and pick again. The loop continues until the vocabulary reaches the target size, which is usually a hyperparameter chosen up front (32K, 50K, 128K, 200K, and so on).

The artifact produced is a learned, ordered list of merges. At encoding time the tokenizer reapplies that list greedily to any input string to produce a sequence of token ids. Both algorithms are deterministic at encoding time and produce one segmentation per input. The split between them is purely about how the winning pair is chosen during training.

BPE: the frequency criterion

WordPiece: the likelihood criterion

Why the difference matters and why it mostly does not

Where they sit in the 2026 stack

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

GPT-5.5 and o3 use BPE via tiktoken's o200k_base encoding with a 200K vocabulary; no WordPiece anywhere in the OpenAI stack in 2026.
Llama 3 and Llama 4 use BPE with a 128K vocabulary, having switched away from the SentencePiece tokenizer used in Llama 2.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhere does the freq(AB) over freq(A) times freq(B) formula come from?

Start from the unigram language model likelihood of the corpus, write the change in log-likelihood from merging pair (A, B) into a new token AB, and show that maximizing that delta reduces to picking the pair with the largest score freq(AB) divided by freq(A) times freq(B). The ratio is the closed-form solution under the unigram independence assumption.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Saying BPE and WordPiece differ in how they segment at inference time. They differ in how they pick merges during training; both apply learned merges greedily at encoding time.

60 second bullets to scan on the way to the call.

State the shared skeleton: iterative pair selection, merge, repeat to a target vocabulary size.
Name BPE's merge criterion in one phrase.

Primary sources. Browse if you want the original framing.