Approximately what fraction of BPE vocabulary tokens does LiteToken identify as 'merge residues'? · Zenaique

Attempt it

Quick reference

TL;DR

LiteToken found that about 10% of BPE vocabulary slots are merge residues: intermediate tokens created during training that almost never appear in real tokenized text.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine building a giant LEGO castle by gluing small bricks into bigger blocks, step by step. Some of those medium-sized blocks were only ever a stepping stone, because you always glued them into something larger, so the medium block never ends up in the finished castle on its own. BPE training works the same way: it merges character pairs into bigger pieces over and over. A chunk of those mid-size pieces are pure stepping stones that the final tokenizer never actually uses. LiteToken counted them and found roughly one in ten vocabulary slots is this kind of leftover, taking up space in the model without ever doing useful work.

Key concepts

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: greedy merge chains + why intermediates die + ~10% residue figure + embedding/lm_head cost + prune and reallocate fix.

Real-world examples

Real products, models, and research that use this idea.

LiteToken (Feb 2026) audited GPT-family and Llama tokenizers and reported roughly 10% of vocabulary tokens are merge residues.
Llama 4 Maverick ships a 128k-class vocabulary, where a 10% residue rate would imply over 12k slots worth pruning and reallocating.
BoundlessBPE (COLM 2025) attacks the same slack from the other side, raising bytes per token via cross word boundary merges.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you empirically identify which specific tokens are residues in a given tokenizer?

Sign in or go Pro to see the approach for this follow-up.

QIf you prune residues, how do you reallocate the freed slots without retraining the whole model?

Sign in or go Pro to see the approach for this follow-up.

QWhy doesn't a Unigram LM tokenizer suffer the same residue problem as BPE?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming every vocabulary slot is actively used because BPE is greedy: in practice roughly 10% are dead-end merge intermediates that never surface in real text.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.

OpenAI tiktoken: BPE tokenizer reference implementation