An LLM never sees raw text. So what does it process?
FlashcardEasyCoinbaseMongodb
Sign in to see the senior framing.
The senior framing: tradeoffs and depth.
Character level tokenization
Word level tokenization
Subword tokenization (BPE, WordPiece, Unigram LM)
Byte level subword tokenization (byte level BPE)
Closed vocabulary; any unseen word becomes [UNK] and information is lost
No native failure mode at this resolution; this is the Pareto compromise modern LLMs use
Latin script bias: non-Latin scripts often need many bytes per character, inflating token counts and costs
Sequence length explosion that blows up attention cost and shrinks effective context
Word level tokenizers fail on new words not seen during training, while character level tokenizers produce sequences far too long for attention to process efficiently.
Subword tokenizers always produce shorter sequences than word level tokenizers for the same input text.
Character level tokenizers require a larger vocabulary than subword tokenizers, making them memory inefficient.
Word level tokenizers use too much memory because each word needs its own embedding vector.
Using a standard GPT style BPE tokenizer on the string:
'hello world'
Approximate token count?
SentencePiece applies NFKC normalization, which additionally decomposes compatibility characters (half width katakana, ligatures, circled numbers) unlike NFC which only handles canonical equivalences.
SentencePiece applies NFC normalization, which is identical to NFKC for all practical purposes since both produce the same byte sequences for standard text.
SentencePiece applies no Unicode normalization and passes raw bytes directly to its segmentation model.
SentencePiece applies NFD normalization, which decomposes composed characters into their base + combining mark form before tokenization.