An LLM never sees raw text. So what does it actually process?

**A token** is the smallest unit a language model reads or writes. It is not a character and it is not a word. It is usually a sub-word chunk of bytes that the tokenizer has assigned an integer id. The flow is simple. Before the model sees your prompt, a tokenizer splits the text into pieces and looks up an id for each piece. The model only ever processes those ids. When it generates a reply, it emits ids, and the tokenizer turns the ids back into text. The model never touches raw characters. The same string can produce different token counts under different tokenizers. The word 'tokenization' is a single token in tiktoken's o200k_base, but it might be three or four tokens in a smaller vocabulary. There is no universal answer to 'how many tokens is this'; it depends on which tokenizer. Why this matters in practice. API pricing is per token. Context window limits are in tokens. Generation speed is tokens per second. Anything you measure about an LLM in dollars, memory, or latency is measured in tokens, not in characters or words.

**A token** is the atomic unit that an LLM reads and emits. It is typically a sub-word chunk of UTF-8 bytes that the tokenizer maps to an integer id drawn from a fixed vocabulary. The model itself is a function from sequences of token ids to sequences of token ids; it never sees raw characters or words. The tokenizer is a separate component that runs before and after the model. On the way in, it splits text into pieces using a learned algorithm (usually byte-level BPE in 2026) and looks up an id for each piece. On the way out, it concatenates the byte sequences for emitted ids back into text. The split between tokenizer and model is strict: the model's embedding matrix and output head are sized to exactly vocab_size, and ids only have meaning relative to one specific tokenizer. Token counts depend on the tokenizer, not the text. 'tokenization' is one token in cl100k_base and o200k_base, but it would be more under a smaller vocabulary. Non-English text and code often tokenize less efficiently than English prose, which is why multilingual workloads cost more per character. Why this matters operationally. Cost, context window, and latency are all measured in tokens. A 100K-token context window holds roughly 75K English words, but far fewer in code-heavy or non-Latin scripts. Anyone budgeting LLM workloads has to think in tokens, not in pages or characters.