An LLM never sees raw text. So what does it actually process?

Attempt it

Your answer

Quick reference

TL;DR

A token is the atomic unit an LLM reads and emits, usually a sub-word chunk of bytes mapped to an integer id by the tokenizer. The model only ever sees token ids, never raw text.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine the language model only understands numbers, not letters. Before the model ever sees your message, a small program called the tokenizer chops the text into little pieces and looks up a number for each piece. Sometimes a piece is a whole short word like 'the'. Sometimes it is half of a long word, like 'token' plus 'ization'. Each piece gets its own number, and the model reads the list of numbers. When the model replies, it produces numbers, and the tokenizer turns them back into letters for you. The pieces are called tokens. They are not characters, they are not words, they sit somewhere in between. The bill you pay and the size of the conversation the model can remember are both measured in tokens, not in characters and not in words.

Key concepts

Token Token ID Subword Unit Tokenizer Context Window

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: state the byte-level definition, place the tokenizer as a separate component before and after the model, contrast tokens against characters and words, show that counts are tokenizer-specific, and link the concept to cost, context window, and latency.

Real-world examples

Real products, models, and research that use this idea.

OpenAI's tiktoken library exposes cl100k_base (GPT-4, GPT-3.5-turbo, text-embedding-ada-002) and o200k_base (GPT-4o, o1, o3, GPT-5, GPT-5.5), letting you count tokens locally before sending a request.
Anthropic's Claude tokenizer is proprietary; the only reliable way to count tokens for Opus 4.7 or Sonnet 4.6 is the messages.count_tokens endpoint.
Llama 3 and Llama 4 ship with a 128K-vocabulary tiktoken-style BPE tokenizer, replacing the 32K SentencePiece tokenizer used in Llama 2.
Hugging Face's tokenizers library (Rust core, Python bindings) is the production standard for training and applying custom tokenizers.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the same text produce different token counts under different tokenizers?

Sign in or go Pro to see the approach for this follow-up.

QIf the model only ever sees token ids, how does it handle a brand-new word it has never seen?

Sign in or go Pro to see the approach for this follow-up.

QWhy do tasks like 'reverse this string' or 'count the letters in apple' confuse modern LLMs?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating tokens as words or as characters. Tokens are sub-word units defined by the tokenizer, and the same string can produce different token counts under different tokenizers.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.