Flashcard: what is a token in LLM prompting and why does it matter for cost and context?

**A token** is the unit of text that an LLM actually reads, writes, and bills you for. It is not a word and it is not a character. A tokenizer splits any input text into a sequence of subword pieces (the tokens), and the model only ever sees that sequence. For English prose the rough conversion is 1 token to about 0.75 words, or roughly 4 characters per token. So 100 tokens cover around 75 words and a one-page document is usually 400 to 600 tokens. Tokens matter for two practical reasons. First, you pay per token. Every API quote shows a price per million input tokens and a separate (usually higher) price per million output tokens. Second, every model has a context-window limit measured in tokens. Claude Opus 4.7, Gemini 3.1, and GPT-5.5 all give you around 1M tokens via the API, and that budget has to cover the system prompt, the user message, the conversation history, any retrieved context, plus the model's reply. Code, numbers, punctuation, and non-English text tokenize denser than plain English, so estimating tokens by word count is unreliable.

**A token** is the subword unit the model operates on. The tokenizer (BPE for most modern models including GPT and Llama 3, SentencePiece for Llama 2 and T5, WordPiece for BERT-family encoders) splits the input string into a sequence of these subword pieces drawn from a fixed vocabulary, usually 50K to 200K entries. The model never sees raw characters; it only ever sees token ids. For English prose, 1 token is roughly 0.75 words or 4 characters. Common words like the are one token; rarer words split into multiple pieces, so antidisestablishmentarianism might be six or seven tokens. Code, numbers, punctuation, emoji, and non-English scripts tokenize less efficiently. A 500-word block of Python or Mandarin can use two to three times more tokens than 500 words of English prose. Two practical consequences make tokens unavoidable. API pricing is per token, with output tokens typically 3 to 5 times more expensive than input tokens, so output length is a leverage point. The context window (the model's maximum total input plus output in a single call) is measured in tokens, not words or characters. Claude Opus 4.7, Gemini 3.1, and GPT-5.5 all ship around 1M tokens via API. Word count is not a reliable proxy for either cost or fit. Always count tokens explicitly with the tokenizer for the model you are using.