BPE and WordPiece both merge subwords. What is the one thing they disagree on?

MCQ·Easy·4.0 · 0·~1 min

Attempt it

**BPE and WordPiece** are both subword tokenizer training algorithms. They both start from a base alphabet (bytes or characters), and they both grow a vocabulary by repeatedly picking two adjacent pieces and merging them into a new piece. They differ in one place: how they pick which pair to merge next. BPE picks the pair that appears most often in the training corpus. Count every adjacent pair across the corpus, pick the pair with the highest count, merge it, repeat. That is the whole algorithm. WordPiece picks the pair whose merge would most increase the likelihood of the corpus under a unigram language model. In practice this works out to maximizing the score freq(AB) divided by freq(A) times freq(B). A pair that occurs often, but where each piece is rare on its own, scores high. A pair where both pieces are already very common scores low, even if the pair itself is frequent. In 2026, BPE is what generative LLMs use: GPT-5.5 (tiktoken cl100k_base and o200k_base), Llama 3 and 4, Mistral. WordPiece is the BERT-family choice and shows up mostly in research and legacy encoder pipelines.

**BPE and WordPiece** are both iterative subword tokenizer training algorithms. They share the same skeleton: start from a base alphabet, count adjacent pairs across the corpus, pick a winning pair, merge it everywhere, repeat until the vocabulary reaches the target size. The headline difference is the scoring function used to pick the winning pair. BPE picks the pair with the highest raw frequency. This is a pure count-based criterion, originally borrowed from data compression (Gage 1994, Sennrich et al. 2016). The selected pair is whichever adjacency shows up most often in the current corpus state. WordPiece picks the pair whose merge most increases corpus likelihood under a unigram language model. The full derivation reduces to picking the pair (A, B) that maximizes freq(AB) divided by freq(A) times freq(B). This ratio is high when AB occurs together far more often than chance would predict given how often A and B occur separately. WordPiece therefore prefers merges that are statistically surprising, not merely common. In 2026 production, BPE dominates generative LLMs: GPT-5.5 uses tiktoken with the o200k_base BPE encoder, Llama 3 and 4 use a 128K BPE vocabulary, Mistral uses a SentencePiece BPE. WordPiece is the BERT-family algorithm and is now mostly research or legacy encoder tooling. Both apply their learned merges greedily at encoding time; the algorithmic split is purely about training.

Quick reference

TL;DR

BPE picks the next merge by raw pair frequency; WordPiece picks the merge that maximally increases corpus likelihood, scored as freq(AB) over freq(A) times freq(B).

Memory aid

Easy to grasp

Picture a kid stacking blocks. Both kids start with a pile of letter blocks and a rule: each turn, glue two blocks together to make a bigger block, and add the new shape to your toolbox. The first kid (BPE) glues whichever two blocks happen to sit next to each other most often. Simple counting. The second kid (WordPiece) is fussier. It only glues two blocks together if the pair shows up much more often than you would expect from how often each block shows up on its own. That ratio rewards merges that are genuinely surprising. After many turns, both kids have a toolbox of bigger blocks they can use to build any word. They tend to pick similar blocks for common words but diverge on rarer combinations, which is why a text tokenized with BPE and the same text tokenized with WordPiece end up looking subtly different.

Key concepts

Byte Pair Encoding WordPiece Subword Tokenization Merge Rule

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: state the shared training skeleton, name BPE's frequency criterion, name WordPiece's likelihood criterion with the freq(AB) over freq(A) times freq(B) ratio, explain why the ratio rewards surprising merges, and place both algorithms in the 2026 production landscape with one or two model examples each.

Real-world examples

Real products, models, and research that use this idea.

GPT-5.5 and o3 use BPE via tiktoken's o200k_base encoding with a 200K vocabulary; no WordPiece anywhere in the OpenAI stack in 2026.
Llama 3 and Llama 4 use BPE with a 128K vocabulary, having switched away from the SentencePiece tokenizer used in Llama 2.
BERT, DistilBERT, and the legacy encoder family still ship with WordPiece tokenizers; this is the main place WordPiece appears in 2026 production code.
The HuggingFace tokenizers library exposes both BPE and WordPiece as first-class training algorithms, which is how the two stay accessible side by side.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhere does the freq(AB) over freq(A) times freq(B) formula come from?

QWhy have most generative LLMs converged on BPE rather than WordPiece?

QIf both produce one segmentation per input, where does any meaningful difference between encoded outputs come from?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Saying BPE and WordPiece differ in how they segment at inference time. They differ in how they pick merges during training; both apply learned merges greedily at encoding time.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.