A token is the atomic unit an LLM reads and emits, usually a sub-word chunk of bytes mapped to an integer id by the tokenizer. The model only ever sees token ids, never raw text.
Imagine the language model only understands numbers, not letters. Before the model ever sees your message, a small program called the tokenizer chops the text into little pieces and looks up a number for each piece. Sometimes a piece is a whole short word like 'the'. Sometimes it is half of a long word, like 'token' plus 'ization'. Each piece gets its own number, and the model reads the list of numbers. When the model replies, it produces numbers, and the tokenizer turns them back into letters for you. The pieces are called tokens. They are not characters, they are not words, they sit somewhere in between. The bill you pay and the size of the conversation the model can remember are both measured in tokens, not in characters and not in words.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Of all the words in the LLM vocabulary, 'token' is the one most worth getting precisely right. Pricing pages, context-window specifications, latency benchmarks, and most prompt-engineering folklore all live in token units. Reasoning about an LLM workload in characters or words quietly produces wrong answers.
This deep dive nails down what a token actually is at the byte level, separates the tokenizer from the model, explains why two tokenizers disagree about the same string, and connects the abstract idea to the operational concerns it drives: cost, capacity, and speed.
The byte-level definition
A token is an integer id between 0 and vocab_size minus 1, drawn from a fixed vocabulary defined by a specific tokenizer. The id points to a particular byte sequence. For modern byte-level BPE tokenizers (tiktoken's cl100k_base and o200k_base, the Llama 3+ tokenizer, Mistral, DeepSeek V4) the byte sequence is a chunk of UTF-8 bytes, typically representing a sub-word fragment of text but sometimes a whole short word or a piece of whitespace.
The id is the only thing the model ever sees. The embedding matrix is indexed by token id: each row is a learned vector for the token at that index. The output projection (lm_head) produces a logit per token id at every position. The model has no separate concept of characters; it works entirely in this discrete vocabulary.
The split between text and ids is strict. On the input side, the tokenizer converts the text string into a list of ids. On the output side, the tokenizer converts the model's emitted ids back into bytes and decodes them as UTF-8 to recover text. The model itself is bytes-blind.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI's tiktoken library exposes cl100k_base (GPT-4, GPT-3.5-turbo, text-embedding-ada-002) and o200k_base (GPT-4o, o1, o3, GPT-5, GPT-5.5), letting you count tokens locally before sending a request.
- Anthropic's Claude tokenizer is proprietary; the only reliable way to count tokens for Opus 4.7 or Sonnet 4.6 is the messages.count_tokens endpoint.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the same text produce different token counts under different tokenizers?
Each tokenizer has its own learned vocabulary and merge rules. A frequent word like 'tokenization' may have its own dedicated id in a large vocabulary but require several merges in a smaller one. Walk through how vocabulary size and training corpus determine which strings get a single id.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating tokens as words or as characters. Tokens are sub-word units defined by the tokenizer, and the same string can produce different token counts under different tokenizers.
60 second bullets to scan on the way to the call.
State in one sentence what a token is at the byte level.
Explain why token count is not the same as character count or word count.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.