An LLM never sees raw text. So what does it process?
FlashcardEasyCoinbaseMongodb
Primary sources. Browse if you want the original framing.
Primary sources behind the answer.
Primary sources. Browse if you want the original framing.
Character level tokenization
Word level tokenization
Subword tokenization (BPE, WordPiece, Unigram LM)
Byte level subword tokenization (byte level BPE)
Closed vocabulary; any unseen word becomes [UNK] and information is lost
No native failure mode at this resolution; this is the Pareto compromise modern LLMs use
Latin script bias: non-Latin scripts often need many bytes per character, inflating token counts and costs
Sequence length explosion that blows up attention cost and shrinks effective context
Primary sources. Browse if you want the original framing.
Word level tokenizers fail on new words not seen during training, while character level tokenizers produce sequences far too long for attention to process efficiently.
Subword tokenizers always produce shorter sequences than word level tokenizers for the same input text.
Character level tokenizers require a larger vocabulary than subword tokenizers, making them memory inefficient.
Word level tokenizers use too much memory because each word needs its own embedding vector.
Primary sources. Browse if you want the original framing.
Using a standard GPT style BPE tokenizer on the string:
'hello world'
Approximate token count?
Primary sources. Browse if you want the original framing.
SentencePiece applies NFKC normalization, which additionally decomposes compatibility characters (half width katakana, ligatures, circled numbers) unlike NFC which only handles canonical equivalences.
SentencePiece applies NFC normalization, which is identical to NFKC for all practical purposes since both produce the same byte sequences for standard text.
SentencePiece applies no Unicode normalization and passes raw bytes directly to its segmentation model.
SentencePiece applies NFD normalization, which decomposes composed characters into their base + combining mark form before tokenization.
Primary sources. Browse if you want the original framing.
Primary sources. Browse if you want the original framing.
Primary sources. Browse if you want the original framing.
Primary sources. Browse if you want the original framing.
Primary sources. Browse if you want the original framing.
Primary sources. Browse if you want the original framing.
Primary sources. Browse if you want the original framing.
Use tiktoken.get_encoding('cl100k_base') and len(encoded), since Claude uses the same BPE family as GPT-4
Call Anthropic's messages.count_tokens endpoint with the formatted message list and read input_tokens from the response
Estimate len(text) / 4 as a rule of thumb, since most production token counts are within 10 percent of this heuristic
Load the deprecated Claude 2 tokenizer from the old @anthropic-ai/tokenizer package and use it for all Claude versions including Sonnet 4.6
Primary sources. Browse if you want the original framing.
Primary sources. Browse if you want the original framing.
Name four common special tokens in modern LLMs and describe what goes wrong if BOS or EOS is missing.
Primary sources. Browse if you want the original framing.