Tokenizer training shapes token count, context efficiency, and representational granularity, so it directly changes pretraining economics and quality.
Imagine writing a long note but deciding where to split words into puzzle pieces first. If you split well, each line carries lots of meaning. If you split badly, the same note needs many extra pieces and takes longer to read. A tokenizer does that splitting for model training. Better splits mean fewer tokens, better use of context, and cleaner learning signals. Bigger pieces are not always better, though. A piece that is too rare shows up so seldom that the reader never learns what it means. And once the pieces are cut, the whole training run uses them, so a bad cut costs you on every page.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A tokenizer turns raw text into the integer IDs that the model trains on. Tokenizer training decides which strings get their own ID. A BPE or unigram algorithm reads a sample corpus and fills a fixed number of vocabulary slots, such as 32,000 or 128,000. After that, the model never sees characters or words. It sees only token sequences, and every cost in pretraining is counted in tokens.
That is why the tokenizer is a first-order choice. It decides how many tokens the corpus becomes, so it sets the compute needed to read the data and how much text fits in each context window. It also decides which patterns arrive as one unit and which arrive as fragments that the model must put back together. Both effects act on every training step. The tokenizer is frozen once pretraining starts, so they last for the life of the model. Option A names both effects: sequence efficiency and representation granularity.
Tokens are the unit of compute
Training compute for a dense transformer is about 6 FLOPs per parameter per training token:
Here N is the parameter count and D is the number of training tokens. D is counted in tokens, not in bytes or words. So the same corpus costs a different amount of compute under a different tokenizer. A tokenizer that packs more characters into each token makes D smaller for the same text.
The Llama 3 tokenizer shows the size of the effect. Its 128K vocabulary raised compression on a sample of English data from 3.17 to 3.94 characters per token, compared with the 32K Llama 2 tokenizer. That is about 20% fewer tokens for the same English text. Put the other way, a fixed token budget now covers about 24% more text.
The saving can be spent in two ways: lower cost on the same data, or more data for the same cost. Either way, the tokenizer moves the budget before any architecture or optimizer choice comes in. Compute-optimal rules such as about 20 tokens per parameter are also stated in tokens. A model trained with a less efficient tokenizer reads less raw text at the same ratio.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3 replaced the 32K Llama 2 tokenizer with a 128K vocabulary (100K tiktoken tokens plus 28K for non-English text) and raised English compression from 3.17 to 3.94 characters per token.
- Codex added tokens for runs of whitespace to the GPT-3 tokenizer, which cut the number of tokens needed to represent code by about 30%.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat tokenizer metric should be tracked before launching a trillion-token run?
Use per-domain tokens per byte and rare-pattern fragmentation, then tie to projected compute and quality risk.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating tokenizer choice as a serving detail ignores that it sets pretraining token budgets and the granularity of what the model learns.
60 second bullets to scan on the way to the call.
Tokenizer as unit of learning interface
Token count and FLOPs connection
Primary sources. Browse if you want the original framing.
- Sennrich et al., Neural Machine Translation of Rare Words with Subword Units
- Kudo and Richardson, SentencePiece: A simple and language independent subword tokenizer
- Llama Team, The Llama 3 Herd of Models
- Ali et al., Tokenizer Choice For LLM Training: Negligible or Crucial?
- Dagan et al., Getting the most out of your tokenizer for pre-training and domain adaptation
Same topic, related formats. Practice these next.