What are BPE merge residues (LiteToken) and what do they imply for vocabulary efficiency?

**BPE merge residues** are leftover tokens from the way Byte Pair Encoding builds its vocabulary, and LiteToken (Feb 2026) found that roughly 10% of a typical BPE vocabulary is made of them. Here is how they arise. BPE builds tokens by merging pairs: token A plus token B becomes a new token C. So A and B had to exist before C was created. But BPE keeps merging, so later it might learn C plus D becomes E. If C almost always ends up merged further into E, then C, and sometimes A and B, rarely appear on their own in real tokenized text. They were necessary stepping stones during training, but they are bypassed at encoding time. These stepping-stone tokens are the residues. The problem is that every token in the vocabulary still gets a row in the embedding matrix and a slot in the output head, whether or not it is ever used. So residues are dead parameters: they cost memory and compute without earning their keep. LiteToken's contribution was to measure this and point out that pruning residues frees that budget to spend on tokens that actually appear.

**Merge residues** are intermediate BPE tokens that the algorithm had to create on the way to building larger tokens. They almost never survive into the final tokenization of real text. LiteToken, from February 2026, measured them and found roughly 10% of a major BPE vocabulary falls into this category. The mechanism is the merge chain. BPE learns ordered merges where A plus B yields C, then C plus D yields E. Each intermediate must exist before the next merge can reference it. But greedy encoding always applies the longest available merge. So if C is nearly always absorbed into E, then C, and even A and B, rarely appear independently in encoded output. They are stranded stepping stones. Why it matters: the vocabulary size sets the embedding matrix rows and the `lm_head` output dimension. Every residue still consumes a row and a logit slot, plus a share of the softmax compute, while contributing essentially nothing to compression. So roughly 10% of that capacity is dead weight. The implication LiteToken draws is vocabulary pruning. Remove or remap residues and reallocate that parameter and compute budget to tokens that genuinely improve bytes per token, sharpening tokenizer efficiency without growing the vocabulary.