Normalization is the preprocessing step the tokenizer runs before pre-tokenization: Unicode normalization, lowercasing, accent stripping, whitespace collapsing. It makes equivalent strings produce identical token ids.
Imagine you can type the word 'cafe' two ways. One way uses a single character with the accent built in. The other way uses a plain 'e' followed by an invisible accent mark. To you, both look identical. To the tokenizer, they are different sequences of bytes and could end up with different token ids. Then the model would treat them as two different words. Normalization is the cleanup step that fixes this: it rewrites both forms into a single canonical form before tokenization even starts. Some normalizers also lowercase everything, strip accents entirely, or collapse repeated whitespace. The exact recipe is part of the tokenizer's configuration and has to match between training and inference. Modern OpenAI tokenizers do almost no normalization; some other tokenizers do a lot.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Normalization is the most invisible part of a tokenizer. It runs before pre-tokenization and merging. It transforms the input string into a canonical form, so small variations in encoding do not produce different token sequences. Most users never think about it. Most production bugs in tokenization pipelines involve it.
Normalization in the tokenizer pipeline
Tokenizer pipelines have four sequential stages: normalization, pre-tokenization, model (BPE merges or SentencePiece encoding), and post-processing.
Normalization runs first. It transforms the input string into a canonical form. The output is a string, not a token sequence.
Pre-tokenization runs second. It splits the normalized string into coarse chunks (typically word-like fragments) using a regex or rule set. The output is a list of strings; the merge algorithm will work within each chunk separately.
Model runs third. For BPE this is the merge application; for SentencePiece-Unigram this is the segmentation search. The output is a list of token ids.
Post-processing runs last. It optionally adds special tokens (BOS, EOS, role markers) around the token sequence.
Normalization is therefore a string transformation. It does not know about tokens, vocabularies, or merges. Its job is to ensure that two semantically equivalent inputs become byte-identical before the rest of the pipeline runs.
The HuggingFace tokenizers library makes this structure explicit. A tokenizer.json file contains a normalizer field that describes the normalizer as a structured config. It says which Unicode form to use, whether to lowercase, whether to strip accents, and what whitespace handling to apply. The tokenizer applies the normalizer first when it encodes any input.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI's tiktoken (cl100k_base, o200k_base) applies almost no normalization and treats input as raw bytes.
- Hugging Face tokenizer.json files for BERT family models include explicit normalizer configurations. bert-base-uncased lowercases and strips accents (NFD, then drops combining marks). bert-base-cased only cleans control characters and spaces out Chinese characters, with no Unicode normalization.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the difference between NFC and NFKC sometimes important for an LLM tokenizer?
NFC only unifies canonical equivalents (different encodings of the same character). NFKC additionally unifies compatibility variants: full-width Latin characters (often appearing in Japanese and Chinese text) become ASCII Latin; ligatures become component letters; super- and sub-scripts become regular characters. NFKC discards information NFC preserves. The choice matters when your inputs include such variants and you want them treated as identical.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming all tokenizers normalize the same way. tiktoken does almost nothing; Hugging Face tokenizers vary widely. Mismatched normalization between training and inference produces silent drift in token sequences.
60 second bullets to scan on the way to the call.
Define tokenizer normalization in one sentence.
Name the four Unicode normalization forms (NFC, NFD, NFKC, NFKD).
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.