BPE is additive and greedy: start from bytes, merge the most frequent pair, repeat, and produce one deterministic segmentation per input. Unigram LM is subtractive and probabilistic: start from a large candidate vocabulary, prune tokens whose removal hurts corpus likelihood least, and produce token probabilities that allow sampling multiple segmentations of the same string.
**BPE and Unigram LM** are two ways to train a subword tokenizer, and they differ in almost every meaningful dimension despite both producing a finite vocabulary of subword pieces.
BPE is additive and greedy. The training loop starts from the base alphabet (bytes or characters), counts every adjacent pair across the corpus, picks the most frequent pair, merges it into a new piece, and repeats until the vocabulary reaches the target size. At encoding time, BPE applies the learned merges greedily and produces exactly one segmentation per input string.
Unigram LM is subtractive and probabilistic. The training loop starts from a large candidate vocabulary that includes many candidate substrings. It then iteratively prunes tokens whose removal would hurt the corpus likelihood least, where likelihood is measured under a unigram language model assumption. Training uses the EM algorithm to refine token probabilities. At encoding time, Unigram LM picks the most probable segmentation by default but can also sample alternative segmentations from the distribution of valid splits.
In 2026, BPE is the dominant generative-LLM choice (GPT-5.5, Llama 3 and 4, Mistral, DeepSeek). Unigram LM lives inside SentencePiece's default mode and was the algorithm behind Llama 2, Gemma, and T5-family models.
**BPE and Unigram LM** are two subword tokenizer training algorithms that differ along three axes: direction, determinism, and output flexibility.
BPE is additive and greedy. Starting from the base alphabet, it repeatedly picks the most frequent adjacent pair, merges it everywhere, and adds it as a new vocabulary entry. The training artifact is an ordered merge list. At encoding time, BPE applies the merges greedily and produces a single deterministic segmentation per input string.
Unigram LM is subtractive and probabilistic. Starting from a large candidate vocabulary (typically built by enumerating frequent substrings), it iteratively prunes tokens to maximize corpus log-likelihood under a unigram model. Training is run via the EM algorithm: the E step computes expected counts of each token given current probabilities, the M step updates probabilities to match those counts, and pruning removes the lowest-utility tokens. The training artifact is a vocabulary with associated token probabilities. At encoding time, Unigram LM picks the maximum-likelihood segmentation by default but supports subword regularization, which samples alternative segmentations from the distribution of valid splits. Sampling at training time gives the downstream model multiple views of the same input string, a free data-augmentation strategy introduced by Kudo in 2018.
In 2026, BPE dominates generative LLMs: GPT-5.5 via tiktoken o200k_base, Llama 3 and 4 with byte-level BPE, Mistral and DeepSeek using BPE-family encoders. Unigram LM is the default in Google's SentencePiece library and was the algorithm behind Llama 2, Gemma, and the T5 family. The choice between BPE and Unigram LM is one of the actual live tokenizer-design decisions in 2026, unlike the mostly-historical BPE-versus-WordPiece split.
Quick reference
TL;DR
BPE grows the vocabulary by merging the most frequent pair (additive, deterministic). Unigram LM shrinks a large candidate vocabulary by pruning the least useful tokens (subtractive, probabilistic, supports sampling).
Imagine two ways to build a Lego set. The first way starts with single bricks. You glue together the two bricks you use together most often, then repeat, until you have a small collection of useful chunks. That is BPE. Each turn adds one new piece. The second way works the other direction. You start with every Lego shape that exists, even strange ones, and you keep throwing out the shape you would miss least when building things. After enough rounds you are left with a smaller box of shapes that still let you build almost anything. That is Unigram LM. One adds, the other prunes. There is another difference. BPE always builds a thing one way. Unigram LM remembers how often each shape gets used, so it can offer alternative builds of the same thing, which is handy when you want to train a model on slightly varied versions of the same input.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Picking a tokenizer to match an existing pretrained model. You do not get to choose the algorithm; the tokenizer is married to the pretrained weights. Mixing tokenizer algorithms with the wrong model corrupts the embedding lookups.
Training a small model where data augmentation is the bottleneck. If you need subword regularization (multiple sampled segmentations per input), Unigram LM is the cleaner choice; BPE-dropout works but is less principled and less widely supported.
Targeting the OpenAI tiktoken ecosystem. tiktoken ships only BPE encoders; if your tooling depends on tiktoken for token counts or for the cl100k_base / o200k_base vocabularies, you are committed to BPE.
Quick scan · 2–4 min
Everything important, quickly.
4 min: name the direction split (additive merging versus subtractive pruning), the optimization objective for each, the default and sampled segmentation behavior, the subword regularization story, and place each algorithm in the 2026 model landscape.
Real-world examples
Real products, models, and research that use this idea.
1Llama 2 used SentencePiece with Unigram LM and a 32K vocabulary; Llama 3 and 4 switched to byte-level BPE with a 128K vocabulary, marking the high-profile generative-LLM move from Unigram to BPE.
2Google's Gemma family uses SentencePiece Unigram LM; T5 and FLAN-T5 also use Unigram LM via SentencePiece.
3OpenAI's tiktoken is byte-level BPE end to end; cl100k_base and o200k_base are both BPE encoders.
4SentencePiece supports both modes via the model_type parameter, so the same tooling can produce a BPE tokenizer or a Unigram LM tokenizer depending on which way you point it.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does Unigram LM use EM rather than a simpler optimization loop?
Sign in or go Pro to see the approach for this follow-up.
QIf both algorithms produce a vocabulary of similar size on the same corpus, why do they encode strings differently?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the analogue of subword regularization for BPE, and why is it less common?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Calling Unigram LM 'just a different scoring function for greedy merges'. It is the opposite direction: Unigram LM starts large and prunes, while BPE starts small and grows. The training procedure and the output behavior both differ.