Predict the output: how many tokens does this string become with a typical BPE tokenizer?
Using a standard GPT style BPE tokenizer on the string: 'hello world' Approximate token count?
Both 'hello' and 'world' are common English words in any GPT-family BPE vocabulary, so the string tokenizes to two tokens, not eleven characters.
Imagine a librarian who has memorized the most common five-thousand English words. When you hand her the phrase 'hello world', she does not spell it out letter by letter; she just recognizes two words she already knows. A BPE tokenizer works the same way. It built up a list of common letter chunks during training, and any whole word that appears often enough in everyday English ends up as its own chunk. 'hello' is one such chunk. The space plus 'world' becomes another single chunk because GPT-style tokenizers treat the leading space as part of the token. The whole string is two tokens, not eleven characters and not three pieces.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Token counting feels like one of those trivia questions that does not deserve a deep dive, until you remember that every cost, latency, and context-budget decision in a production LLM pipeline runs off this number. Underestimate by 30 percent and your monthly bill quietly grows; overestimate and you reject prompts that would have fit.
This question, 'hello world' under a GPT-style BPE tokenizer, has a clean answer (two tokens) and a surprising amount of structure underneath. The structure is what an interviewer is probing for. The exact count is the floor; the reasoning is the ceiling.
Why 'hello' and 'world' are single tokens
BPE builds its vocabulary by counting adjacent symbol pairs in a training corpus and merging the most frequent pair, repeatedly, until it hits the target vocabulary size. The English words hello and world rank among the few thousand most common tokens in any web-scale English corpus, so their character sequences get merged into single vocabulary entries very early in the BPE training process.
The result is that any modern OpenAI tokenizer (cl100k_base, o200k_base) maps each of these words to one integer. You can verify this in two lines of Python with the tiktoken library, and the same logic applies to Anthropic, Google, and Meta tokenizers for these specific words because they all train on broadly similar English corpora.
The non-obvious bit. Capitalization can matter: hello and Hello are usually separate tokens in the vocabulary because both forms appear often enough in training data to earn their own entries. But they are still one token each.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI billing on the GPT-5.5 and o-series APIs is computed in tokens via tiktoken; underestimating token counts is the most common cost-projection bug.
- Anthropic's Claude Opus 4.7 and Sonnet 4.x ship a related but distinct tokenizer; the same string can have different counts across providers.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat would change if the string were 'Hello, world!' instead?
The comma and exclamation split off as their own tokens, and the capital H still maps to a single token, so the count rises to four or five.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Counting characters or assuming the leading space splits off as its own token. Modern GPT-style BPE bundles a leading space with the following word.
60 second bullets to scan on the way to the call.
Why common English words become single tokens in modern BPE
How GPT-style pre-tokenization handles the leading space
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.