Before the tokenizer even starts splitting, normalization runs. What does it do and why?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

Normalization is a text preprocessing step the tokenizer applies before pre-tokenization or merging: Unicode normalization (NFC or NFKC), lowercasing, accent stripping, whitespace collapsing. Its job is to guarantee that visually identical strings encoded differently in Unicode produce the same token ids. Modern Hugging Face tokenizer.json files declare the normalizer explicitly; tiktoken applies almost no normalization and is essentially raw byte-level.

Your answer

**Normalization** is the text-preprocessing step the tokenizer applies before pre-tokenization or merging. Its job is to clean up text variations that should not matter for the model. The most common operations. **Unicode normalization** (NFC or NFKC) rewrites the input into a canonical form, so the same visible character is always represented the same way at the byte level. **Lowercasing** converts everything to lowercase, used by uncased models like bert-base-uncased. **Accent stripping** removes diacritics, less common in modern LLMs. **Whitespace collapsing** turns multiple spaces, tabs, or newlines into a single space. Why this matters. The same word 'cafe' with an accent can be encoded two ways in Unicode: as a single precomposed character (one byte sequence) or as the base letter 'e' plus a combining accent (a different byte sequence). Without normalization, the tokenizer would treat these as different tokens. Normalization unifies them into one canonical form. Modern OpenAI tokenizers (cl100k_base, o200k_base) apply almost no normalization. They operate close to raw bytes and let the model handle small variations. Hugging Face tokenizers expose the normalizer as an explicit configuration in tokenizer.json, and the choice varies widely per model. The normalization recipe has to match between training and inference; using a different one at inference time produces silent token-sequence drift.

**Normalization** is the first stage of the tokenizer pipeline, running before pre-tokenization and BPE merging. Its purpose is to make logically equivalent inputs produce identical byte sequences before they reach the encoding algorithm. The standard operations. **Unicode normalization** is the foundation. Unicode allows the same visual character to be encoded multiple ways: 'cafe' with a precomposed accented 'e' (U+00E9) versus 'cafe' with a base 'e' (U+0065) followed by a combining acute accent (U+0301). Without normalization, these produce different byte sequences and different tokens. NFC (Normalization Form Canonical Composition) prefers the composed form; NFKC (Compatibility Composition) goes further and unifies compatibility variants (full-width Latin to ASCII, ligatures to component letters). **Lowercasing** is used by uncased models. **Accent stripping** removes diacritics; was common in older NLP but uncommon in modern LLMs. **Whitespace handling** can collapse runs, strip leading and trailing, or replace special whitespace with regular spaces. The normalization recipe is part of the tokenizer's contract. It is applied identically at training and at inference; mismatched normalization is a silent bug that produces different token sequences for the same logical input. Hugging Face tokenizer.json files declare the normalizer explicitly as a structured config. tiktoken applies almost no normalization, treating input as raw bytes and pushing the disambiguation responsibility downstream. In 2026 the trend across frontier LLMs is toward minimal normalization. Byte-level BPE is robust to small input variation by construction (alternate encodings still encode losslessly, just to different token sequences), and frontier model teams prefer preserving input information rather than collapsing it. cl100k_base, o200k_base, and the Llama 3+ tokenizer all do very little normalization. Older models (BERT, T5, the original SentencePiece-based Llama 2) do more.

Quick reference

TL;DR

Normalization is the preprocessing step the tokenizer runs before pre-tokenization: Unicode normalization, lowercasing, accent stripping, whitespace collapsing. It makes equivalent strings produce identical token ids.

Memory aid

Easy to grasp

Imagine the same word 'cafe' can be typed two different ways on your keyboard: one way uses a single special character with the accent baked in, and the other way uses the plain letter 'e' followed by an invisible accent mark. To you, both look identical. To the tokenizer, they are different sequences of bytes and could end up with different token ids, which means the model would treat them as two different words. Normalization is the cleanup step that fixes this: it rewrites both forms into a single canonical form before tokenization even starts. Some normalizers also lowercase everything, strip accents entirely, or collapse repeated whitespace. The exact recipe is part of the tokenizer's configuration and has to match between training and inference. Modern OpenAI tokenizers do almost no normalization; some other tokenizers do a lot.

Key concepts

Normalization NFC NFKC Unicode Preprocessing

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define normalization as the first stage of the tokenizer pipeline, name the four Unicode normalization forms, explain why NFC matters for byte identity, contrast tiktoken's minimal normalization with Hugging Face's explicit normalizer config, and warn that training and inference normalization must match.

Real-world examples

Real products, models, and research that use this idea.

OpenAI's tiktoken (cl100k_base, o200k_base) applies essentially no normalization, treating input as raw bytes; this is documented in the tiktoken source.
Hugging Face tokenizer.json files for BERT family models include explicit normalizer configurations: bert-base-uncased applies lowercase, NFD, and strip_accents; bert-base-cased applies NFD only.
SentencePiece tokenizers (used by older Llama 2, T5, Gemini) replace whitespace with U+2581 and apply NFKC by default; the byte_fallback setting interacts with normalization to determine whether unusual characters can be encoded losslessly.
The Hugging Face tokenizers library exposes pre-built normalizers (NFC, NFD, NFKC, NFKD, Lowercase, Strip, StripAccents, Replace, Sequence) that can be composed into a custom normalizer chain.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the difference between NFC and NFKC sometimes important for an LLM tokenizer?

QIf tiktoken applies almost no normalization, how does GPT-5 handle the two encodings of cafe case?

QWhat happens if you fine-tune a model with a different normalization than the base model was trained with?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming all tokenizers normalize the same way. tiktoken does almost nothing; Hugging Face tokenizers vary widely. Mismatched normalization between training and inference produces silent drift in token sequences.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.