The answer is **NFKC**. Unicode normalization is a cleanup step that rewrites text into a canonical form before tokenizing, so that visually identical strings map to the same bytes.
There are four forms. NFC and NFD handle canonical equivalence, things that are defined to be the same character, like an accented `é` written as one codepoint versus `e` plus a combining accent. NFKC and NFKD go further: they also fold compatibility characters, such as the `fi` ligature, half-width katakana, circled numbers, and superscripts, into plain equivalents.
BERT's tokenizer applies NFKC, which means it does the canonical cleanup and the compatibility cleanup. NFC only does the canonical part. So for any text containing a ligature or a half-width character, NFKC and NFC produce different bytes, and therefore different token IDs. That difference is the source of a lot of silent multilingual bugs.
BERT applies **NFKC**, and the distinction from NFC is the compatibility decomposition.
Think of the forms in two tiers. Canonical normalization, used by NFC and NFD, only unifies characters Unicode defines as truly equivalent, like the two ways to write an accented letter. Compatibility normalization, the K in NFKC and NFKD, additionally folds characters that merely look or function the same. Examples include ligatures (`fi` becomes `fi`), half-width katakana into full-width, circled digits like `①` into `1`, and superscripts into their base digits.
Because NFKC rewrites those characters and NFC does not, the two forms produce different byte sequences for any input containing compatibility characters. After tokenizing, that means different token IDs.
The interview-relevant consequence is the train/inference mismatch. If BERT was trained on NFKC text but a serving pipeline forwards NFC text, the model sees byte patterns it never trained on for those characters. It still runs, no error, just quietly degraded behavior on Japanese, math, and formatted text.
Quick reference
TL;DR
BERT normalizes with NFKC, which folds compatibility characters like ligatures and half-width katakana to canonical forms, whereas NFC leaves them untouched.
Imagine two people typing the word 'office'. One uses a fancy joined-up 'fi' that their font offers as a single decorative glyph; the other types plain 'f' then 'i'. To a human both look the same, but to a computer they are different characters. NFKC is the cleanup step that says 'treat the decorative version exactly like the plain letters'. It also flattens shrunk-down katakana, circled numbers, and superscripts into their ordinary forms. NFC is a lighter cleanup: it tidies up accents but leaves the decorative versions alone. BERT uses the heavier NFKC cleanup, so two inputs that merely differ in those decorative characters end up identical before tokenizing.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Your input is guaranteed plain ASCII or simple Latin text with no ligatures or width variants. NFC and NFKC agree on this subset, so the heavier compatibility folding buys nothing and only adds preprocessing cost.
You need to preserve the exact distinction between full-width and half-width forms, such as faithful Japanese typography. NFKC collapses width distinctions, destroying information that some downstream tasks actually depend on.
You are reproducing another model that was trained without normalization or with a different form. Matching the original tokenizer's normalization matters more than picking the theoretically cleaner form, since mismatch degrades behavior.
Quick scan · 2–4 min
Everything important, quickly.
3 min: four normalization forms + canonical vs compatibility + NFKC folds ligatures/half-width + train/serve mismatch as silent bug.
Code example
python
import unicodedata
# The fi ligature (U+FB01) and a circled digit (U+2460)
s = "o\uFB01ce \u2460"
# NFC keeps compatibility characters as-is
nfc = unicodedata.normalize("NFC", s)
# NFKC folds them to base forms: ligature -> 'fi', circled-1 -> '1'
nfkc = unicodedata.normalize("NFKC", s)
print(nfc == nfkc) # False: they diverge on compatibility chars
print(repr(nfc), repr(nfkc)) # 'o\ufb01ce \u2460' vs 'office 1'
# A BERT-style tokenizer trained on NFKC must be fed NFKC at inference,
# or token IDs for these characters silently diverge from training.
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face BERT tokenizers apply NFKC via the BertNormalizer, so half-width katakana and ligatures fold before WordPiece runs.
2Multilingual search pipelines normalize queries with NFKC so that a half-width and full-width form of the same Japanese term hit the same index entries.
3PDF-to-text extraction in document RAG stacks often emits ligatures, and an NFKC pass prevents 'office' from tokenizing differently than typed text.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect a normalization mismatch between training and serving after the model is already deployed?
Sign in or go Pro to see the approach for this follow-up.
QWhy might NFKC hurt rather than help for a faithful Japanese typing assistant?
Sign in or go Pro to see the approach for this follow-up.
QDo byte-level BPE models like GPT need normalization the way WordPiece does?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming NFC and NFKC are interchangeable because they agree on plain ASCII: they diverge on any ligature, half-width, or circled character, which is exactly where multilingual bugs hide.