The short takeaway for each question.
A token is the atomic unit an LLM reads and emits, usually a sub-word chunk of bytes mapped to an integer id by the tokenizer. The model only ever sees token ids, never raw text.
Character level tokenization
Word level tokenization
Subword tokenization (BPE, WordPiece, Unigram LM)
Byte level subword tokenization (byte level BPE)
Closed vocabulary; any unseen word becomes [UNK] and information is lost
No native failure mode at this resolution; this is the Pareto compromise modern LLMs use
Latin script bias: non-Latin scripts often need many bytes per character, inflating token counts and costs
Sequence length explosion that blows up attention cost and shrinks effective context
Character-level explodes sequence length. Word-level hits closed vocabulary, so unseen words become [UNK]. Subword is the Pareto winner LLMs use; byte-level still has a Latin-script bias.
Word level tokenizers fail on new words not seen during training, while character level tokenizers produce sequences far too long for attention to process efficiently.
Subword tokenizers always produce shorter sequences than word level tokenizers for the same input text.
Character level tokenizers require a larger vocabulary than subword tokenizers, making them memory inefficient.
Word level tokenizers use too much memory because each word needs its own embedding vector.
Subword tokenization is the compromise that handles unseen words by splitting them into known pieces while keeping sequences short, avoiding both word-level OOV failures and character-level length explosion.
Using a standard GPT style BPE tokenizer on the string:
'hello world'
Approximate token count?
Both 'hello' and 'world' are common English words in any GPT-family BPE vocabulary, so the string tokenizes to two tokens, not eleven characters.
SentencePiece applies NFKC normalization, which additionally decomposes compatibility characters (half width katakana, ligatures, circled numbers) unlike NFC which only handles canonical equivalences.
SentencePiece applies NFC normalization, which is identical to NFKC for all practical purposes since both produce the same byte sequences for standard text.
SentencePiece applies no Unicode normalization and passes raw bytes directly to its segmentation model.
SentencePiece applies NFD normalization, which decomposes composed characters into their base + combining mark form before tokenization.
SentencePiece normalizes with NFKC by default, which folds compatibility characters like ligatures and half-width katakana to plain forms, whereas NFC leaves them untouched.
Normalization is the preprocessing step the tokenizer runs before pre-tokenization: Unicode normalization, lowercasing, accent stripping, whitespace collapsing. It makes equivalent strings produce identical token ids.
Pre-tokenization is the regex step that splits raw text into coarse chunks before BPE merging. It enforces hard boundaries: merges can never cross them, which shapes what the vocabulary can contain.
A tokenizer vocabulary is the fixed numbered list of token ids the tokenizer can emit. Its size is set at training time and directly determines the row count of the embedding matrix and the column count of the lm_head.
Each tokenizer builds its own vocabulary independently, so id 1234 is a row index into one specific table. Different tokenizer, different table, different meaning. Raw text is the only portable representation.
Bigger vocab compresses text into shorter sequences (cheaper attention) but fattens the embedding matrix, lm_head, and softmax. The sweet spot for general LLMs is 100K to 200K.
In it: tokenizer class, special token ids, model_max_length, padding_side, chat_template. NOT in it: vocabulary mappings and merge rules, which live in tokenizer.json or a .model file.
Use tiktoken.get_encoding('cl100k_base') and len(encoded), since Claude uses the same BPE family as GPT-4
Call Anthropic's messages.count_tokens endpoint with the formatted message list and read input_tokens from the response
Estimate len(text) / 4 as a rule of thumb, since most production token counts are within 10 percent of this heuristic
Load the deprecated Claude 2 tokenizer from the old @anthropic-ai/tokenizer package and use it for all Claude versions including Sonnet 4.6
Anthropic does not publish Claude's tokenizer; use the messages.count_tokens API for official counts. tiktoken, len/4 heuristics, and the deprecated Claude 2 tokenizer are all wrong for Claude 3 and later.
Reserved ids prevent ambiguity: BOS, EOS, and PAD occupy positions normal tokenization never produces, so they can never collide with real content.
Name four common special tokens in modern LLMs and describe what goes wrong if BOS or EOS is missing.
BOS starts generation, EOS stops it (runs to the length cap if missing), PAD aligns batches and must be attention-masked, and role markers fence off who speaks in a chat.