Predict the output: how many tokens does this string become with a typical BPE tokenizer?
Using a standard GPT-style BPE tokenizer on the string: 'hello world' Approximate token count?
Both 'hello' and 'world' are common English words in any GPT-family BPE vocabulary, so the string tokenizes to two tokens, not eleven characters.
Imagine a librarian who has memorized the most common five-thousand English words. When you hand her the phrase 'hello world', she does not spell it out letter by letter; she just recognizes two words she already knows. A BPE tokenizer works the same way. It built up a list of common letter chunks during training, and any whole word that appears often enough in everyday English ends up as its own chunk. 'hello' is one such chunk. The space plus 'world' becomes another single chunk because GPT-style tokenizers treat the leading space as part of the token. The whole string is two tokens, not eleven characters and not three pieces.
Detailed answer & concept explanation~4 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
3 min: count for 'hello world' + GPT space-attachment rule + four chars per token heuristic + when it breaks (code, JSON, non-English) + provider differences.
Real products, models, and research that use this idea.
- OpenAI billing on the GPT-5.5 and o-series APIs is computed in tokens via tiktoken; underestimating token counts is the most common cost-projection bug.
- Anthropic's Claude Opus 4.7 and Sonnet 4.x ship a related but distinct tokenizer; the same string can have different counts across providers.
- Meta's Llama 4 uses a 128k-entry tokenizer, so the same prompt is typically slightly more compact than with cl100k_base.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat would change if the string were 'Hello, world!' instead?
QHow would you estimate token cost for a 50k-line code repository?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Counting characters or assuming the leading space splits off as its own token. Modern GPT-style BPE bundles a leading space with the following word.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.