Standard

A token is the smallest unit that an LLM reads and emits. It is usually a sub-word chunk of UTF-8 bytes that the tokenizer maps to an integer id drawn from a fixed vocabulary. The model itself is a function from sequences of token ids to sequences of token ids; it never sees raw characters or words.

The tokenizer is a separate component that runs before and after the model. On the way in, it splits text into pieces using a learned algorithm (usually byte-level BPE) and looks up an id for each piece. On the way out, it concatenates the byte sequences for emitted ids back into text.

The split between tokenizer and model is strict. The model's embedding matrix and output head are sized to exactly vocab_size, and ids only have meaning relative to one specific tokenizer.

Token counts depend on the tokenizer, not the text. 'embedding' is one token in cl100k_base and o200k_base, but two tokens in GPT-2's smaller vocabulary. Non-English text and code often tokenize less efficiently than English prose, which is why multilingual workloads cost more per character.

Cost, context window, and latency are all measured in tokens. A 100K-token context window holds roughly 75K English words, but far fewer in code or in non-Latin scripts. Anyone budgeting LLM workloads has to think in tokens, not in pages or characters.

Three tokenization families have three failure modes, and a hybrid family wins in production.

Character-level makes every character (or byte) its own token. The vocabulary is tiny and there is no unknown input. The dominant failure mode is sequence length. The same paragraph that fits in 200 subword tokens may need 1200 character tokens. Attention is O(n^2) in sequence length, so character-level costs several times more to train and run. It also shrinks the effective context window by the same factor.

Word-level splits on whitespace and treats each word as one token. Sequences are short and tokens carry clean meaning. The dominant failure mode is the closed vocabulary. Any word not in the training vocabulary becomes [UNK], and the original information is lost. New product names, typos, and rare jargon collapse onto [UNK].

Subword tokenization (BPE, WordPiece, Unigram LM) is the Pareto compromise. Pieces fall between characters and whole words. Byte-level variants have no out of vocabulary. Sequence length is reasonable (roughly four characters per token under o200k_base). Every generative LLM uses subword tokenization: GPT-4o and Llama 3 use byte-level BPE, and Mistral 7B and Gemma use SentencePiece BPE.

Byte-level subword still has a real failure mode: a Latin-script bias. UTF-8 encodes ASCII in one byte but many non-Latin code points in two, three, or four bytes. The tokenizer learns its merges on mostly English data. Non-Latin scripts (Chinese, Arabic, Hindi, Korean) often need more tokens per character than English, inflating cost and shrinking effective context.

LLMs use subword tokenization because it resolves the two opposite failures of word-level and character-level approaches at once.

Word-level tokenization builds a closed dictionary at training time. Any word missing from it (a new name or a typo) maps to a single [UNK] token and loses all information. That is the out of vocabulary problem.

Character-level tokenization never has OOV, since anything can be spelled from characters, but it explodes sequence length. A five character word becomes five tokens. Attention cost scales with the square of length, written O(n^2). So a roughly 5x longer sequence means about 25x more attention computation, which is impractical to train.

Subword tokenization takes the middle path. It merges frequent patterns into single tokens, keeping common text short. It splits rare or unseen words into known subpieces, so there is no OOV. Typical English lands near 1.3 tokens per word, close to word-level efficiency, while still representing any string. That Pareto balance (open vocabulary plus tractable length) is why BPE and its variants are the standard.

Two tokens. hello and world (note the leading space attached to the second word) are both common English forms. GPT-style BPE tokenizers assign each to a single vocabulary entry.

BPE merges adjacent symbols during training in order of frequency. Common English words like hello and world get merged early and stay as single tokens in the final vocabulary. The leading space is bundled into the following word because GPT pre-tokenization keeps whitespace as part of the next token. That is why world and world are different tokens, but each is still one token, not two.

A broader rule of thumb is roughly four characters per token for English text. The string hello world is 11 characters, so 2 to 3 tokens is the right ballpark. Run tiktoken if you ever need the exact count.

SentencePiece applies NFKC by default, and the distinction from NFC is the compatibility decomposition.

Think of the forms in two tiers. Canonical normalization, used by NFC and NFD, only unifies characters Unicode defines as truly equivalent, like the two ways to write an accented letter. Compatibility normalization, the K in NFKC and NFKD, additionally folds characters that merely look or function the same. Examples include ligatures (fi becomes fi), half-width katakana into full-width, circled digits like ① into 1, and superscripts into their base digits.

Because NFKC rewrites those characters and NFC does not, the two forms produce different byte sequences for any input containing compatibility characters. After tokenizing, that means different token IDs.

The key consequence is the train/inference mismatch. Suppose a model was trained on NFKC text but the serving path tokenizes with a port that applies NFC. Then the model sees byte patterns it never trained on for those characters. It still runs with no error, but behavior on Japanese, math, and formatted text quietly degrades.

Normalization is the first stage of the tokenizer pipeline, running before pre-tokenization and BPE merging. Its purpose is to make logically equivalent inputs produce identical byte sequences before they reach the encoding algorithm.

Unicode normalization is the foundation. Unicode allows the same visual character to be encoded multiple ways. For example, 'cafe' can use a precomposed accented 'e' (U+00E9) or a base 'e' (U+0065) followed by a combining acute accent (U+0301). Without normalization, these produce different byte sequences and different tokens.

NFC (Normalization Form Canonical Composition) prefers the composed form. NFKC (Compatibility Composition) goes further and unifies compatibility variants (full-width Latin to ASCII, ligatures to component letters).

Lowercasing is used by uncased models. Accent stripping removes diacritics, common in older NLP but rare in modern LLMs. Whitespace handling can collapse runs, strip leading and trailing, or replace special whitespace with regular spaces.

The normalization recipe is part of the tokenizer's contract. The tokenizer applies it identically at training and at inference. Mismatched normalization is a silent bug that produces different token sequences for the same logical input. Hugging Face tokenizer.json files declare the normalizer explicitly as a structured config. tiktoken applies almost no normalization and treats input as raw bytes.

The trend across frontier LLMs is toward minimal normalization. Byte-level BPE is robust to small input variation by construction: alternate encodings still encode losslessly, just to different token sequences. cl100k_base, o200k_base, and the Llama 3+ tokenizer all do very little normalization. Older models (BERT, T5, Llama 2) do more.