Which statement correctly describes the Unigram Language Model tokenizer's training procedure?

Attempt it

Quick reference

TL;DR

Unigram LM starts with a large vocabulary and prunes the tokens whose removal least hurts corpus likelihood, the opposite direction from BPE's additive merges.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine you start with a giant box holding every word and word-fragment you could ever use. Your job is to shrink the box to a fixed size, keeping only the most useful pieces. Instead of guessing, you test each piece: if I throw this one out, how much harder is it to describe my training text? You keep tossing the pieces that you barely miss, and stop when the box is the right size. That is Unigram LM. It prunes a big vocabulary down by usefulness, which is the reverse of the other popular method that starts tiny and glues frequent pieces together.

Key concepts

Unigram Language Model EM algorithm Corpus log-likelihood Subtractive vocabulary building

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: subtractive vs additive + EM E-step and M-step + likelihood objective + pruning by smallest drop + probabilistic output enables sampling.

Real-world examples

Real products, models, and research that use this idea.

Google's T5 and mT5 use SentencePiece Unigram LM, where likelihood-based pruning balances vocabulary across many languages.
The Llama-family SentencePiece tokenizers were built with the Unigram algorithm, giving a probabilistic segmentation model rather than pure BPE merges.
ALBERT ships a SentencePiece Unigram tokenizer, relying on its sampling to regularize subword segmentation during pretraining.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the E-step use forward-backward over all segmentations rather than just the Viterbi best path?

Sign in or go Pro to see the approach for this follow-up.

QHow does stochastic tokenization from Unigram act as data augmentation, and what robustness does it buy?

Sign in or go Pro to see the approach for this follow-up.

QHow would you choose the target vocabulary size and the per-iteration pruning fraction?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Mixing up the directions. Unigram LM prunes a large vocabulary down by likelihood, while BPE merges up from characters by frequency.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.