Normalization is the preprocessing step the tokenizer runs before pre-tokenization: Unicode normalization, lowercasing, accent stripping, whitespace collapsing. It makes equivalent strings produce identical token ids.
Imagine the same word 'cafe' can be typed two different ways on your keyboard: one way uses a single special character with the accent baked in, and the other way uses the plain letter 'e' followed by an invisible accent mark. To you, both look identical. To the tokenizer, they are different sequences of bytes and could end up with different token ids, which means the model would treat them as two different words. Normalization is the cleanup step that fixes this: it rewrites both forms into a single canonical form before tokenization even starts. Some normalizers also lowercase everything, strip accents entirely, or collapse repeated whitespace. The exact recipe is part of the tokenizer's configuration and has to match between training and inference. Modern OpenAI tokenizers do almost no normalization; some other tokenizers do a lot.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Normalization is the most invisible part of a modern tokenizer. It runs before pre-tokenization and merging, transforming the input string into a canonical form so that small variations in encoding do not produce different token sequences. Most users never think about it. Most production bugs in tokenization pipelines involve it.
This deep dive defines normalization precisely, names the standard operations, explains why Unicode normalization specifically matters, and walks through where it has gone in modern LLM tokenizers (mostly: away).
Normalization in the tokenizer pipeline
Modern tokenizer pipelines have four sequential stages: normalization, pre-tokenization, model (BPE merges or SentencePiece encoding), and post-processing.
Normalization runs first. It transforms the input string into a canonical form. The output is a string, not a token sequence.
Pre-tokenization runs second. It splits the normalized string into coarse chunks (typically word-like fragments) using a regex or rule set. The output is a list of strings; the merge algorithm will work within each chunk separately.
Model runs third. For BPE this is the merge application; for SentencePiece-Unigram this is the segmentation search. The output is a list of token ids.
Post-processing runs last. It optionally adds special tokens (BOS, EOS, role markers) around the token sequence.
Normalization is therefore a string transformation. It does not know about tokens, vocabularies, or merges. Its job is to ensure that two semantically equivalent inputs become byte-identical before the rest of the pipeline runs.
The HuggingFace tokenizers library makes this structure explicit. A tokenizer.json file contains a normalizer field that describes the normalizer as a structured config: which Unicode form, whether to lowercase, whether to strip accents, what whitespace handling to apply. The normalizer is applied first when the tokenizer encodes any input.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI's tiktoken (cl100k_base, o200k_base) applies essentially no normalization, treating input as raw bytes; this is documented in the tiktoken source.
- Hugging Face tokenizer.json files for BERT family models include explicit normalizer configurations: bert-base-uncased applies lowercase, NFD, and strip_accents; bert-base-cased applies NFD only.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the difference between NFC and NFKC sometimes important for an LLM tokenizer?
NFC only unifies canonical equivalents (different encodings of the same character). NFKC additionally unifies compatibility variants: full-width Latin characters (often appearing in Japanese and Chinese text) become ASCII Latin; ligatures become component letters; super- and sub-scripts become regular characters. NFKC discards information NFC preserves. The choice matters when your inputs include such variants and you want them treated as identical.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming all tokenizers normalize the same way. tiktoken does almost nothing; Hugging Face tokenizers vary widely. Mismatched normalization between training and inference produces silent drift in token sequences.
60 second bullets to scan on the way to the call.
Define tokenizer normalization in one sentence.
Name the four Unicode normalization forms (NFC, NFD, NFKC, NFKD).
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.