Follow ups

Each tokenizer has its own learned vocabulary and merge rules. A frequent word like 'embedding' has its own dedicated id in a large vocabulary (o200k_base) but splits into two pieces in a smaller one (GPT-2). Walk through how vocabulary size and training corpus determine which strings get a single id.

Each English character is roughly one byte, so character-level produces about four times the tokens of a 100K-200K subword vocabulary on English. Attention is O(n^2) in sequence length, so 4x tokens means 16x attention compute per layer. KV cache size also scales linearly with sequence length, so memory is 4x. The effective context window for a fixed compute budget shrinks by the same multiplier.

Every string reduces to bytes, and the vocabulary always includes all 256 byte values, so any input decomposes into in-vocabulary pieces.

The comma and exclamation split off as their own tokens, and the capital H still maps to a single token, so the count rises to four or five.

Run both forms over a corpus, diff the token IDs, and flag inputs where they differ; concentrate testing on Japanese, math, and PDF-extracted text.

NFC only unifies canonical equivalents (different encodings of the same character). NFKC additionally unifies compatibility variants: full-width Latin characters (often appearing in Japanese and Chinese text) become ASCII Latin; ligatures become component letters; super- and sub-scripts become regular characters. NFKC discards information NFC preserves. The choice matters when your inputs include such variants and you want them treated as identical.