Compare the Unigram Language Model tokenizer (used in SentencePiece) to BPE in terms of: (1) training direction (additive vs. subtractive), (2) the optimization objective, and (3) the key capability that Unigram's probabilistic output enables during model training.
Free · 2 AI evals / day
**BPE** and **Unigram LM** are the two main subword tokenizer training algorithms, and they work in opposite directions.
BPE starts from the smallest possible vocabulary, individual characters or bytes, and grows. At each step it finds the most frequent adjacent pair and merges it into a new token, repeating until the vocabulary reaches the target size. BPE always produces exactly one tokenization of any string.
Unigram LM starts from a large candidate vocabulary, every n-gram up to some length plus common subwords, and shrinks. At each step it works out which tokens contribute least to describing the training text, measured by corpus likelihood, and removes them. It repeats until the vocabulary reaches the target size.
The special property of Unigram LM is that it learns a probability for every possible split of a string, not just one. So the same word can be cut differently on different training passes. That built-in variation is a kind of data augmentation, making the model sturdier on spelling and morphology variation, which BPE cannot offer because it is deterministic.
**BPE is additive**, starting from characters and merging up by frequency. **Unigram LM is subtractive**, starting from a large vocabulary and pruning down by likelihood.
Unigram LM frames vocabulary selection as likelihood maximization through an EM loop. The E-step estimates token probabilities across all possible segmentations of the training sentences. The M-step prunes the tokens whose removal costs the least likelihood. The objective sums the corpus log-likelihood over every segmentation, treating the split as a hidden variable.
BPE, by contrast, has a single criterion: merge the most frequent pair, irreversibly.
The capability this unlocks is **stochastic tokenization**. Because Unigram LM yields a distribution over segmentations, training can sample different splits of the same word. For example, 'tokenization' becomes token plus ization on one pass and token, iz, ation on another. That is subword-level data augmentation, which improves robustness to typos, morphology, and out of distribution text. At inference you switch back to the single most probable split for reproducibility. BPE produces one deterministic segmentation and cannot do this.
Quick reference
TL;DR
BPE merges up from characters by frequency; Unigram LM prunes down from a large vocabulary by corpus likelihood, and its probabilistic output enables stochastic tokenization as training-time augmentation.
Imagine two ways to settle on a useful set of word-pieces. One way starts with single letters and keeps gluing the most popular pair together, like 'th' then 'the'. The other starts with a huge pile of every chunk imaginable and keeps asking, which chunk would I miss the least, then throws it out, until the pile is the right size.
The second way, Unigram LM, has a bonus. It remembers that a word can be cut in several valid ways and even how likely each cut is. During training it uses different cuts on different passes, which teaches the model to handle the same word no matter how it is sliced. The first way always cuts the same way every time.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A pipeline needing byte-level coverage so no input is ever unknown Plain Unigram operates on characters, so byte-level BPE is safer when arbitrary Unicode must always encode without an unknown token.
Reproducibility-critical serving where every call must match The sampling benefit is training-only; at inference you fix the Viterbi path, so Unigram offers no advantage there over a deterministic encoder.
Reusing a GPT-family BPE vocabulary for ecosystem compatibility Switching to Unigram changes the vocabulary and IDs, breaking compatibility with models and tools built on the BPE tokenizer.
Tiny or extremely noisy training corpora Sparse counts make the EM likelihood estimates unstable, so pruning can drop useful tokens and the augmentation benefit shrinks.
Latency-bound fine-tuning where canonical splits must be preserved Heavy sampling drifts from the pretraining segmentation; keep alpha low or off so the model sees splits close to what it learned.
Quick scan · 2–4 min
Everything important, quickly.
5 min: additive vs subtractive + likelihood objective over segmentations + E-step forward-backward + M-step prune + stochastic tokenization + alpha and inference determinism.
Code example
python
import sentencepiece as spm
sp = spm.SentencePieceProcessor()
sp.load("llama_tokenizer.model") # a Unigram model
text = "tokenization"
# Deterministic Viterbi path: same result every call
print("viterbi:", sp.encode(text, out_type=str))
# Stochastic sampling: different valid splits across calls
for _ in range(4):
print("sample:", sp.encode(text, out_type=str,
enable_sampling=True, alpha=0.2, nbest_size=-1))
Real-world examples
Real products, models, and research that use this idea.
1The Llama-family SentencePiece tokenizers use Unigram, whose probabilistic segmentation supports stochastic tokenization during pretraining.
2Google's T5 and mT5 use SentencePiece Unigram, where likelihood-based pruning balances vocabulary coverage across many languages.
3For agglutinative languages like Finnish and Turkish, sampling different splits of a shared root builds representations robust to unseen derived forms, an edge over deterministic BPE.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QDerive the M-step probability update for a token given expected counts from the E-step.
Sign in or go Pro to see the approach for this follow-up.
QHow does the per-iteration pruning fraction affect final vocabulary quality, say 50% versus 10%?
Sign in or go Pro to see the approach for this follow-up.
QWhat does the alpha parameter in sample_encode control, and how would you set it for pretraining versus fine-tuning?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reversing the directions. BPE is additive and merges up; Unigram LM is subtractive and prunes down. They reach the same vocabulary size from opposite ends.