A token is the model's subword unit of text, roughly 0.75 English words, and it is the unit both API pricing and context-window limits are measured in.
Imagine a typewriter that can only press one of fifty thousand pre-cut stamps instead of individual letters. Each stamp is a token. Some stamps are whole common words like the or apple, others are partial pieces like tion or ing, and others are single characters for rare symbols. To write any sentence the typewriter has to find the right sequence of stamps. When you ask an LLM a question, the model first turns your text into a sequence of these stamps, then predicts the next stamp, then the next, then the next, until it stops. The price you pay is per stamp pressed, both for reading your input and writing its reply. The maximum number of stamps the typewriter can hold in memory at once is the context window.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Tokens are the most under-appreciated unit of LLM engineering. They look like an implementation detail until your first production bill arrives or your first long-context call truncates silently in the middle of a conversation. Every cost decision, every context-budgeting decision, and a surprising amount of quality behavior traces back to how the tokenizer chops up your text.
This deep dive defines what a token actually is, walks through the practical math of counting tokens and quoting cost, explains why non-English and code text uses more tokens, and connects tokenization back to the context-window budget that every real LLM application has to plan around.
What a token actually is
A token is a subword piece produced by a deterministic tokenizer. The tokenizer is trained alongside the model: it scans a large text corpus, finds the most frequent character sequences, and builds a fixed vocabulary of usually 50K to 200K entries. Common words like the and apple often get their own dedicated token. Rarer or longer words split into multiple subword pieces. The word antidisestablishmentarianism, for example, becomes six or seven tokens depending on the vocabulary.
Three tokenization algorithms dominate modern LLMs. BPE (Byte-Pair Encoding) is used by OpenAI's GPT family, Llama 3, and many other modern open-weight models. SentencePiece is used by Llama 2, T5, and some multilingual models. WordPiece is older and survives mainly in BERT-family encoders.
The practical consequence is that every string has exactly one token sequence under a given tokenizer, but that sequence differs across tokenizers. The token count of the same prompt under Claude's tokenizer is not the same as under GPT-5.5's tokenizer, even though both are roughly in the same ballpark for English prose.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic publishes a tokenizer endpoint so teams can count tokens exactly before sending a Claude Opus 4.7 request.
- OpenAI's tiktoken library is the standard way to count tokens locally for GPT-5.5 and GPT-4o prompts without making a network call.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy are output tokens typically 3 to 5 times more expensive than input tokens?
Output is autoregressive, one token at a time, so each output token requires a full forward pass through the model. Input is processed in parallel and benefits from KV-cache reuse across requests when prompt caching is on.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Estimating token count by multiplying word count by 1.0 and being surprised when a code-heavy or non-English prompt blows the context-window budget; tokens are not words.
60 second bullets to scan on the way to the call.
What a token actually is (subword unit from a fixed-vocabulary tokenizer)
Rough conversion rate from tokens to English words (and why it is only a rough rate)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.