A token is the model's subword unit of text, roughly 0.75 English words, and it is the unit both API pricing and context-window limits are measured in.
Imagine a typewriter that can only press one of fifty thousand pre-cut stamps instead of individual letters. Each stamp is a token. Some stamps are whole common words like "the" or "apple", others are partial pieces like "tion" or "ing", and others are single characters for rare symbols. To write any sentence the typewriter has to find the right sequence of stamps. When you ask an LLM a question, the model first turns your text into a sequence of these stamps. Then it predicts the next stamp, then the next, then the next, until it stops. The price you pay is per stamp pressed, both for reading your input and writing its reply. The maximum number of stamps the typewriter can hold in memory at once is the context window.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Tokens are the most underappreciated unit of LLM engineering. They look like an implementation detail until your first production bill arrives or your first long-context call fails with a context-length error. Every cost decision, every context budgeting decision, and a surprising amount of quality behavior traces back to how the tokenizer chops up your text.
What a token is
A token is a subword piece produced by a deterministic tokenizer. The tokenizer is trained before the model, as a separate step. It scans a large text corpus, finds the most frequent character sequences, and builds a fixed vocabulary of usually about 30K to 250K entries. Common words like "the" and "apple" often get their own dedicated token. Rarer or longer words split into multiple subword pieces. The word antidisestablishmentarianism, for example, becomes six or seven tokens depending on the vocabulary.
Three tokenizer families dominate modern LLMs. BPE (Byte-Pair Encoding) is used by OpenAI's GPT family, Llama 3, and many other modern open-weight models. SentencePiece, a library that implements BPE or the Unigram algorithm, is used by Llama 2, T5, and some multilingual models. WordPiece is older and survives mainly in BERT-family encoders.
The practical consequence is that every string has exactly one token sequence under a given tokenizer, but that sequence differs across tokenizers. The token count of the same prompt under Claude's tokenizer is not the same as under GPT-4o's tokenizer. Even so, both are roughly in the same ballpark for English prose.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic provides a token counting API endpoint so teams can count a Claude request's tokens before sending it.
- OpenAI's tiktoken library is the standard way to count tokens locally for GPT-4o and GPT-5 prompts (both use o200k_base) without making a network call.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy are output tokens typically 4 to 8 times more expensive than input tokens?
Output is autoregressive, one token at a time, so each output token requires a full forward pass through the model. Input is processed in parallel and benefits from KV-cache reuse across requests when prompt caching is on.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Estimating token count by multiplying word count by 1.0 and being surprised when a code-heavy or non-English prompt blows the context-window budget; tokens are not words.
60 second bullets to scan on the way to the call.
What a token is (subword unit from a fixed-vocabulary tokenizer)
Rough conversion rate from tokens to English words (and why it is only a rough rate)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.