BERT normalizes with NFKC, which folds compatibility characters like ligatures and half-width katakana to canonical forms, whereas NFC leaves them untouched.
Imagine two people typing the word 'office'. One uses a fancy joined-up 'fi' that their font offers as a single decorative glyph; the other types plain 'f' then 'i'. To a human both look the same, but to a computer they are different characters. NFKC is the cleanup step that says 'treat the decorative version exactly like the plain letters'. It also flattens shrunk-down katakana, circled numbers, and superscripts into their ordinary forms. NFC is a lighter cleanup: it tidies up accents but leaves the decorative versions alone. BERT uses the heavier NFKC cleanup, so two inputs that merely differ in those decorative characters end up identical before tokenizing.
Detailed answer & concept explanation~6 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
3 min: four normalization forms + canonical vs compatibility + NFKC folds ligatures/half-width + train/serve mismatch as silent bug.
import unicodedata
# The fi ligature (U+FB01) and a circled digit (U+2460)
s = "o\uFB01ce \u2460"
# NFC keeps compatibility characters as-is
nfc = unicodedata.normalize("NFC", s)
# NFKC folds them to base forms: ligature -> 'fi', circled-1 -> '1'
nfkc = unicodedata.normalize("NFKC", s)
print(nfc == nfkc) # False: they diverge on compatibility chars
print(repr(nfc), repr(nfkc)) # 'o\ufb01ce \u2460' vs 'office 1'
# A BERT-style tokenizer trained on NFKC must be fed NFKC at inference,
# or token IDs for these characters silently diverge from training.Real products, models, and research that use this idea.
- Hugging Face BERT tokenizers apply NFKC via the BertNormalizer, so half-width katakana and ligatures fold before WordPiece runs.
- Multilingual search pipelines normalize queries with NFKC so that a half-width and full-width form of the same Japanese term hit the same index entries.
- PDF-to-text extraction in document RAG stacks often emits ligatures, and an NFKC pass prevents 'office' from tokenizing differently than typed text.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect a normalization mismatch between training and serving after the model is already deployed?
QWhy might NFKC hurt rather than help for a faithful Japanese typing assistant?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Assuming NFC and NFKC are interchangeable because they agree on plain ASCII: they diverge on any ligature, half-width, or circled character, which is exactly where multilingual bugs hide.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.