Unigram LM starts with a large vocabulary and prunes the tokens whose removal least hurts corpus likelihood, the opposite direction from BPE's additive merges.
Imagine you start with a giant box holding every word and word-fragment you could ever use. Your job is to shrink the box to a fixed size, keeping only the most useful pieces. Instead of guessing, you test each piece: if I throw this one out, how much harder is it to describe my training text? You keep tossing the pieces that you barely miss, and stop when the box is the right size. That is Unigram LM. It prunes a big vocabulary down by usefulness, which is the reverse of the other popular method that starts tiny and glues frequent pieces together.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Unigram LM tokenizer training inverts the usual intuition about building a vocabulary. Instead of discovering tokens by merging frequent pairs, the way BPE does, it selects tokens by asking which candidates best explain the corpus under a probabilistic model, then removes the ones that do not earn their place.
This question rewards spotting both halves of that idea: the direction of training, large to small, and the objective, likelihood rather than frequency. The three wrong options each swap one of those halves for something plausible, so getting it right means you can tell BPE, a supervised segmenter, and a fewest-tokens heuristic apart from the real algorithm.
We will walk the subtractive loop, write down the objective the EM steps optimize, show why the probabilistic output is the feature that BPE structurally cannot match, and finish by dissecting exactly why each distractor is wrong so the distinction sticks.
The subtractive training loop
Unigram LM begins with a large candidate vocabulary, often every character n-gram up to length 6 to 8 plus frequent subwords, which can be tens of thousands of candidates. From there it shrinks. The initial set is deliberately over-complete: it includes far more pieces than the final vocabulary will hold, so the algorithm has room to discover which ones actually carry their weight.
Each round is two steps. The E-step holds the vocabulary fixed and estimates each token's probability so as to maximize the corpus likelihood, using the forward-backward algorithm to spread credit across all possible segmentations of each sentence. The M-step then asks, for every token, how much the corpus likelihood would fall if that token were deleted, and prunes the chunk with the smallest impact, typically 10 to 20% at a time. Pruning in small batches matters, because the likelihood impact of removing one token is only an estimate that ignores interactions, so taking small steps keeps that approximation honest.
The loop repeats until the vocabulary reaches the target size. Note what this is not: there is no merging, no neural classifier, and no fewest-tokens heuristic. The single organizing principle is keep the tokens the corpus relies on, drop the ones it can do without. Characters are never pruned, since they form the irreducible base that guarantees any string can still be segmented after the larger pieces are removed.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Google's T5 and mT5 use SentencePiece Unigram LM, where likelihood-based pruning balances vocabulary across many languages.
- The Llama-family SentencePiece tokenizers were built with the Unigram algorithm, giving a probabilistic segmentation model rather than pure BPE merges.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the E-step use forward-backward over all segmentations rather than just the Viterbi best path?
Training needs expected token counts across every segmentation, not one path; using Viterbi alone would bias the probability estimates toward the current best split.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Mixing up the directions. Unigram LM prunes a large vocabulary down by likelihood, while BPE merges up from characters by frequency.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.