Which statement correctly describes the Unigram Language Model tokenizer's training procedure?

The correct answer is the first option: **Unigram LM** starts with a large candidate vocabulary and prunes it, using an EM-style procedure that keeps the tokens most useful for explaining the corpus. Think of the direction. BPE, in option B, does the opposite: it starts from individual characters and merges the most frequent pair over and over until the vocabulary hits its target size. Unigram LM starts from the other end, with every character n-gram and common subword, then shrinks down. The shrinking is not random. At each round it scores how much each token helps describe the training text, then removes a chunk of the least useful ones, around 10 to 20% at a time. The other two options are wrong: there is no neural boundary classifier here, and it does not just pick the split with the fewest tokens. The defining idea is likelihood-based pruning from a large starting vocabulary.

The answer is the first option. **Unigram LM** is a subtractive, likelihood-based algorithm, the inverse of BPE's additive, frequency-based merging. Training starts with a large candidate vocabulary, typically all character n-grams up to some length plus common subwords. Then it runs an EM loop. The E-step fixes the vocabulary and estimates token probabilities that maximize the corpus likelihood across all possible segmentations. The M-step scores how much removing each token would cost in likelihood and prunes the least useful chunk. This repeats until the vocabulary reaches the target size. The distractors map to real but different ideas. Option B is exactly BPE. Option C describes a learned neural segmenter, not Unigram LM. Option D, fewest-tokens selection, throws away the probabilistic model that is the whole point. The payoff of the probabilistic model: Unigram LM assigns a probability to every tokenization of a string, not just one. That enables sampling different segmentations during training, which BPE cannot do. SentencePiece implements Unigram LM and ships it for models like T5 and the Llama family.