Beginner

A token is the smallest unit a language model reads or writes. It is not a character and it is not a word. It is usually a sub-word chunk of bytes that the tokenizer has assigned an integer id.

The flow is simple. Before the model sees your prompt, a tokenizer splits the text into pieces and looks up an id for each piece. The model only ever processes those ids. When it generates a reply, it emits ids, and the tokenizer turns the ids back into text. The model never touches raw characters.

The same string can produce different token counts under different tokenizers. The word 'embedding' is a single token in tiktoken's o200k_base, but two tokens in GPT-2's smaller vocabulary. There is no universal answer to 'how many tokens is this'; it depends on which tokenizer.

API pricing is per token. Context window limits (how much text fits at once) are in tokens. Generation speed is tokens per second. Anything you measure about an LLM in dollars, memory, or latency is measured in tokens, not in characters or words.

There are three classic families of tokenization, and each has a dominant weakness.

Character-level breaks text into individual characters (or bytes). The vocabulary is tiny (a few hundred entries), and there is never an unknown character.

The problem is sequence length. Every word becomes many tokens, so a paragraph that fits in 200 subword tokens might take 1200 character tokens. Attention cost grows with the square of the sequence length, so this is expensive. It also greatly shrinks the effective context window.

Word-level splits text at whitespace. Sequences are short, and each token carries clear meaning. The problem is the closed vocabulary. Any word outside the vocabulary becomes a special [UNK] token, and the information in the original word is lost. New product names, typos, and rare technical terms all map to [UNK].

Subword (BPE, WordPiece, Unigram LM) splits text into pieces between characters and whole words. Modern variants are byte-level, so there is never an unknown input. Sequence length is reasonable. Every generative LLM uses it.

Byte-level subword still has one quirk: a Latin-script bias. Non-Latin characters often encode to multiple bytes in UTF-8, so Chinese, Arabic, or Hindi text uses more tokens per character than English does.

Tokenization is how text becomes the integer pieces a model reads, and there are three styles. Word-level uses whole words, character-level uses single characters, and subword uses pieces in between.

Word-level has a fixed dictionary built at training time. Any word not in that dictionary, like a new product name, cannot be represented and gets replaced by a single unknown marker, written [UNK]. The model loses all information about that word. This is the out of vocabulary, or OOV, problem.

Character-level fixes OOV completely, because you can always spell anything from characters. But it makes sequences very long. A five letter word becomes five tokens, so a sentence that was ten tokens becomes around fifty. That matters because the cost of attention (the step comparing every token pair) grows with the square of sequence length. Long sequences get expensive fast.

Subword is the practical middle ground. Common words and pieces get their own single token, which keeps sequences short. Rare or new words split into known smaller pieces, which avoids OOV. So subword handles anything you throw at it without exploding the length. That is why LLMs use it.

Tokenization is the step that turns a string into the integer IDs the model reads. The number of tokens is rarely the same as the number of characters or even the number of words.

GPT-style BPE tokenizers (BPE merges frequent character pairs) include OpenAI's cl100k_base and o200k_base vocabularies. They learn their merges from a large English corpus (a big collection of text). Words that appear frequently end up as single tokens. hello and world are common enough to get their own entries in any vocabulary trained on English.

GPT-style tokenizers include the leading space in the token. So the string hello world becomes the tokens hello and world (with the space attached). That is still two tokens, not three. The rough rule of thumb that English averages about four characters per token holds here: 11 characters and roughly 2 to 3 tokens.

SentencePiece, a widely used tokenizer library, uses NFKC by default. Unicode normalization is a cleanup step that rewrites text into a canonical form before tokenizing, so that visually identical strings map to the same bytes.

There are four forms. NFC and NFD handle canonical equivalence: things that are defined to be the same character. An example is an accented é written as one codepoint versus e plus a combining accent. NFKC and NFKD go further: they also fold compatibility characters, such as the fi ligature, half-width katakana, circled numbers, and superscripts, into plain equivalents.

A SentencePiece tokenizer with default settings applies NFKC, which means it does the canonical cleanup and the compatibility cleanup. NFC only does the canonical part. So for any text containing a ligature or a half-width character, NFKC and NFC produce different bytes, and therefore different token IDs. That difference is the source of a lot of silent multilingual bugs.

Normalization is the text preprocessing step the tokenizer applies before pre-tokenization or merging. Its job is to clean up text variations that should not matter for the model.

The most common operations follow. Unicode normalization (NFC or NFKC) rewrites the input into a canonical (single standard) form. Then the same visible character always has the same bytes. Lowercasing converts everything to lowercase, used by uncased models like bert-base-uncased. Accent stripping removes diacritics, less common in modern LLMs. Whitespace collapsing turns multiple spaces, tabs, or newlines into a single space.

The same word 'cafe' with an accent can be encoded two ways in Unicode. One is a single precomposed character (one byte sequence). The other is the base letter 'e' plus a combining accent (a different byte sequence). Without normalization, the tokenizer would treat these as different tokens.

Modern OpenAI tokenizers (cl100k_base, o200k_base) apply almost no normalization. They operate close to raw bytes and let the model handle small variations. Hugging Face tokenizers expose the normalizer as an explicit configuration in tokenizer.json, and it varies widely per model. The normalization recipe has to match between training and inference. Using a different one at inference time produces silent drift in token sequences.