BERT normalizes with NFKC, which folds compatibility characters like ligatures and half-width katakana to canonical forms, whereas NFC leaves them untouched.
Imagine two people typing the word 'office'. One uses a fancy joined-up 'fi' that their font offers as a single decorative glyph; the other types plain 'f' then 'i'. To a human both look the same, but to a computer they are different characters. NFKC is the cleanup step that says 'treat the decorative version exactly like the plain letters'. It also flattens shrunk-down katakana, circled numbers, and superscripts into their ordinary forms. NFC is a lighter cleanup: it tidies up accents but leaves the decorative versions alone. BERT uses the heavier NFKC cleanup, so two inputs that merely differ in those decorative characters end up identical before tokenizing.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Normalization is the least glamorous stage of a tokenizer, which is exactly why it produces such durable bugs. Before WordPiece or BPE ever runs, the raw text is rewritten into a canonical form so that strings a human would call identical map to identical bytes. The question of which form a tokenizer uses sounds like trivia until a multilingual model starts quietly underperforming on Japanese and nobody can find an error in the logs.
BERT applies NFKC. To understand why that matters, you have to separate two notions Unicode keeps deliberately distinct: canonical equivalence and compatibility equivalence. NFC handles only the first. NFKC handles both. That single extra step is the entire difference, and it is the seam where train versus inference mismatches creep in.
This deep dive walks the four forms, the precise NFC-versus-NFKC divergence, and the production failure mode that makes this an interview-worthy topic rather than a footnote.
The four normalization forms, briefly
Unicode defines four normalization forms, built from two independent choices. The first choice is decompose then recompose (composed, the C forms) versus decompose and stop (decomposed, the D forms). The second choice is canonical only versus canonical plus compatibility (the K forms).
That gives NFC and NFD on the canonical side, and NFKC and NFKD on the compatibility side. Canonical normalization unifies characters Unicode declares truly equivalent, the classic example being an accented letter written as a single precomposed codepoint versus a base letter plus a combining accent. Both spellings are the same character; canonical forms make them byte-identical.
The K forms add compatibility folding. Compatibility characters are ones that are not strictly the same character but are close enough that treating them identically is usually desirable: ligatures, half-width and full-width variants, circled and superscript digits.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import unicodedata
# The fi ligature (U+FB01) and a circled digit (U+2460)
s = "o\uFB01ce \u2460"
# NFC keeps compatibility characters as-is
nfc = unicodedata.normalize("NFC", s)
# NFKC folds them to base forms: ligature -> 'fi', circled-1 -> '1'
nfkc = unicodedata.normalize("NFKC", s)
print(nfc == nfkc) # False: they diverge on compatibility chars
print(repr(nfc), repr(nfkc)) # 'o\ufb01ce \u2460' vs 'office 1'
# A BERT-style tokenizer trained on NFKC must be fed NFKC at inference,
# or token IDs for these characters silently diverge from training.Real products, models, and research that use this idea.
- Hugging Face BERT tokenizers apply NFKC via the BertNormalizer, so half-width katakana and ligatures fold before WordPiece runs.
- Multilingual search pipelines normalize queries with NFKC so that a half-width and full-width form of the same Japanese term hit the same index entries.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect a normalization mismatch between training and serving after the model is already deployed?
Run both forms over a corpus, diff the token IDs, and flag inputs where they differ; concentrate testing on Japanese, math, and PDF-extracted text.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming NFC and NFKC are interchangeable because they agree on plain ASCII: they diverge on any ligature, half-width, or circled character, which is exactly where multilingual bugs hide.
60 second bullets to scan on the way to the call.
The four Unicode normalization forms and which fold compatibility characters
What the K in NFKC and NFKD stands for
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.