Compare the Unigram Language Model tokenizer (used in SentencePiece) to BPE in terms of: (1) training direction (additive vs. subtractive), (2) the optimization objective, and (3) the key capability that Unigram's probabilistic output enables during model training.
BPE merges up from characters by frequency; Unigram LM prunes down from a large vocabulary by corpus likelihood, and its probabilistic output enables stochastic tokenization as training-time augmentation.
Imagine two ways to settle on a useful set of word-pieces. One way starts with single letters and keeps gluing the most popular pair together, like 'th' then 'the'. The other starts with a huge pile of every chunk imaginable and keeps asking, which chunk would I miss the least, then throws it out, until the pile is the right size. The second way, Unigram LM, has a bonus. It remembers that a word can be cut in several valid ways and even how likely each cut is. During training it uses different cuts on different passes, which teaches the model to handle the same word no matter how it is sliced. The first way always cuts the same way every time.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The Unigram Language Model tokenizer takes a fundamentally different route to a vocabulary than BPE. BPE is a greedy frequency-counting algorithm that builds the vocabulary bottom-up by merging. Unigram LM is a probabilistic pruning algorithm that selects tokens by how well they explain the corpus, and it converges top-down.
The distinction is worth getting precise because it shows up in two places: the training objective, and a capability at model-training time that BPE simply cannot offer. Most candidates can recite that BPE merges and Unigram prunes. The senior signal is being able to say what each one optimizes, why the EM E-step needs forward-backward rather than the best path, and what the resulting distribution buys you during pretraining.
We will compare the two directions, write down the latent-variable likelihood that the EM steps optimize, explain the forward-backward requirement, show how the probabilistic output becomes stochastic tokenization, and then connect all of it to where each tokenizer actually ships in production.
Two directions to the same vocabulary
BPE initializes with single characters, or bytes in the byte-level variant, and grows by merging the most frequent adjacent pair into a new token. Each merge is a greedy, irreversible local decision that ignores how it affects other possible segmentations. Once 'th' and then 'the' are merged, those choices are locked, and later merges build on top of them whether or not they were globally optimal.
Unigram LM initializes with a large candidate vocabulary, typically all n-grams up to length 6 to 8 from the corpus plus common longer pieces, which can be tens of thousands of candidates. It then removes the tokens whose deletion least hurts the corpus likelihood, iterating down to the target size.
Both reach the same final vocabulary size, but from opposite ends. BPE grows from below by frequency; Unigram LM shrinks from above by likelihood. That difference in selection criterion, not just direction, is what gives Unigram a more globally informed vocabulary. Frequency is a local statistic: it tells you a pair co-occurs often, but not whether keeping it as one token actually helps the model describe the corpus once overlapping tokens compete. Likelihood is a global statistic: it scores a token by its marginal contribution given everything else in the vocabulary. That is why two tokens that each look frequent can still be redundant, and Unigram can notice the redundancy where BPE cannot.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import sentencepiece as spm
sp = spm.SentencePieceProcessor()
sp.load("llama_tokenizer.model") # a Unigram model
text = "tokenization"
# Deterministic Viterbi path: same result every call
print("viterbi:", sp.encode(text, out_type=str))
# Stochastic sampling: different valid splits across calls
for _ in range(4):
print("sample:", sp.encode(text, out_type=str,
enable_sampling=True, alpha=0.2, nbest_size=-1))Real products, models, and research that use this idea.
- The Llama-family SentencePiece tokenizers use Unigram, whose probabilistic segmentation supports stochastic tokenization during pretraining.
- Google's T5 and mT5 use SentencePiece Unigram, where likelihood-based pruning balances vocabulary coverage across many languages.
What an interviewer would ask next. Try answering before peeking at the approach.
QDerive the M-step probability update for a token given expected counts from the E-step.
Maximizing the likelihood reduces to normalizing expected counts, so p(t) equals the expected count of t over the sum of all expected counts; contrast with BPE's raw frequency.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reversing the directions. BPE is additive and merges up; Unigram LM is subtractive and prunes down. They reach the same vocabulary size from opposite ends.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.