Easy to grasp

Imagine the language model only understands numbers, not letters. Before the model ever sees your message, a small program called the tokenizer chops the text into little pieces and looks up a number for each piece. Sometimes a piece is a whole short word like 'the'. Sometimes it is half of a long word, like 'token' plus 'ization'. Each piece gets its own number, and the model reads the list of numbers. When the model replies, it produces numbers, and the tokenizer turns them back into letters for you. The pieces are called tokens. They are not characters and not words. They sit somewhere in between. The bill you pay and the size of the conversation the model can remember are both measured in tokens, not in characters and not in words.

Imagine three ways to pack a backpack for a trip. The first packer (character) puts each item in its own tiny zip-loc bag. Nothing is missing, but the backpack is now full of bags, and it takes forever to find anything. The second packer (word) only packs items from a fixed packing list. The list does not include the new pair of shoes you just bought, so the shoes get left at home. The third packer (subword) packs in medium chunks: a 'shoe' bag, a 'lace' bag, a 'sole' bag. You can build anything new out of those chunks, and the backpack stays a reasonable size. LLMs use packer three. Packer one is too slow and bulky; packer two cannot handle anything new. Even packer three has a quirk. With non-English scripts, each character can take several bytes, so the chunks for those languages end up costing more.

Imagine you have a box of building blocks. With word blocks, you can only build things you have a block for, so a brand-new shape you have never seen is impossible to make. With single-dot blocks, you can build anything, but even a tiny shape needs hundreds of dots and takes forever. Subword blocks are medium-sized pieces. Common shapes have their own ready-made block, and anything new you can still build from smaller pieces. That way you can always make the shape, and you do not need a mountain of tiny dots to do it.

Imagine a librarian who has memorized the most common five-thousand English words. When you hand her the phrase 'hello world', she does not spell it out letter by letter; she just recognizes two words she already knows. A BPE tokenizer works the same way. It built up a list of common letter chunks during training, and any whole word that appears often enough in everyday English ends up as its own chunk. 'hello' is one such chunk. The space plus 'world' becomes another single chunk because GPT-style tokenizers treat the leading space as part of the token. The whole string is two tokens, not eleven characters and not three pieces.

Imagine two people typing the word 'office'. One uses a fancy joined-up 'fi' that their font offers as a single decorative glyph. The other types plain 'f' then 'i'. To a human both look the same, but to a computer they are different characters. NFKC is the cleanup step that says 'treat the decorative version exactly like the plain letters'. It also flattens shrunk-down katakana, circled numbers, and superscripts into their ordinary forms. NFC is a lighter cleanup: it tidies up accents but leaves the decorative versions alone. SentencePiece uses the heavier NFKC cleanup by default. So two inputs that differ only in those decorative characters end up identical before tokenizing.

Imagine you can type the word 'cafe' two ways. One way uses a single character with the accent built in. The other way uses a plain 'e' followed by an invisible accent mark. To you, both look identical. To the tokenizer, they are different sequences of bytes and could end up with different token ids. Then the model would treat them as two different words. Normalization is the cleanup step that fixes this: it rewrites both forms into a single canonical form before tokenization even starts. Some normalizers also lowercase everything, strip accents entirely, or collapse repeated whitespace. The exact recipe is part of the tokenizer's configuration and has to match between training and inference. Modern OpenAI tokenizers do almost no normalization; some other tokenizers do a lot.