Progressive hints when you are stuck.
Stuck? Peek. Progressively more specific. Each peek narrows the answer further.
The model is a function from integers to integers. Think about what those integers represent and at what level of granularity they live.
It is not characters and it is not words. Consider what unit sits in between and why API pricing pages talk about it instead of word count.
Character level tokenization
Word level tokenization
Subword tokenization (BPE, WordPiece, Unigram LM)
Byte level subword tokenization (byte level BPE)
Closed vocabulary; any unseen word becomes [UNK] and information is lost
No native failure mode at this resolution; this is the Pareto compromise modern LLMs use
Latin script bias: non-Latin scripts often need many bytes per character, inflating token counts and costs
Sequence length explosion that blows up attention cost and shrinks effective context
Stuck? Peek. Progressively more specific. Each peek narrows the answer further.
Each family trades one problem for another. Think about what happens when the unit of tokenization is very small (each piece carries little meaning) versus very large (you run out of pieces for new inputs).
Sequence length, vocabulary coverage, and per-script byte cost are the three axes. Map each family to whichever axis hurts it the most.
Word level tokenizers fail on new words not seen during training, while character level tokenizers produce sequences far too long for attention to process efficiently.
Subword tokenizers always produce shorter sequences than word level tokenizers for the same input text.
Character level tokenizers require a larger vocabulary than subword tokenizers, making them memory inefficient.
Word level tokenizers use too much memory because each word needs its own embedding vector.
Stuck? Peek. Progressively more specific. Each peek narrows the answer further.
Think about what happens when a word-level vocabulary trained before 2022 encounters the string 'ChatGPT'. What does the tokenizer emit, and why is that a problem?
Consider both failure directions: word-level hits a wall with out of vocabulary strings, while character-level sidesteps that wall but pays a sequence-length price. How does that price relate to attention's computational cost as sequence length grows?
Using a standard GPT style BPE tokenizer on the string:
'hello world'
Approximate token count?
Stuck? Peek. Progressively more specific. Each peek narrows the answer further.
Both 'hello' and 'world' are common enough that they each fit in one token.
Remember that the leading space on the second word usually merges with that word into a single token.
SentencePiece applies NFKC normalization, which additionally decomposes compatibility characters (half width katakana, ligatures, circled numbers) unlike NFC which only handles canonical equivalences.
SentencePiece applies NFC normalization, which is identical to NFKC for all practical purposes since both produce the same byte sequences for standard text.
SentencePiece applies no Unicode normalization and passes raw bytes directly to its segmentation model.
SentencePiece applies NFD normalization, which decomposes composed characters into their base + combining mark form before tokenization.
Stuck? Peek. Progressively more specific. Each peek narrows the answer further.
NFKC stands for 'Normalization Form Compatibility Composition.' The 'K' stands for 'Compatibility.' What does compatibility decomposition do that canonical decomposition does not?
Think about specific character categories that NFKC but not NFC transforms: ligatures like fi, circled characters like ①, half-width vs. full-width forms. Which tokenizer would produce different token IDs for input text containing these characters: SentencePiece with default settings (NFKC) or GPT-series byte-level BPE (none)?
Stuck? Peek. Progressively more specific. Each peek narrows the answer further.
The same visible character can be encoded in more than one way in Unicode. Think about what should happen if a user types the accented 'cafe' two different ways.
Normalization runs before the tokenizer's main work. Consider the cost of skipping it (training and inference see different bytes) versus the cost of doing it (losing information about case or accents).
Stuck? Peek. Progressively more specific. Each peek narrows the answer further.
BPE merges scan the input looking for the next adjacent pair to combine. Think about what would happen if there were no constraints and the algorithm could merge across whitespace or punctuation.
Pre-tokenization is the step that decides which sub-strings the merge algorithm is even allowed to consider. Consider the regex that splits 'hello, world' into chunks and what merges become impossible as a result.
Stuck? Peek. Progressively more specific. Each peek narrows the answer further.
The vocabulary is the set of things the model can ever say. Think about which model weights depend on its size and what changes when you make it bigger or smaller.
Two large matrices in the model have a dimension equal to the vocabulary size. Identify both, and consider why this means you cannot change the vocabulary without re-training.
Stuck? Peek. Progressively more specific. Each peek narrows the answer further.
Token ids are integers, but they are not arbitrary integers. They have a specific job inside the model. Think about what those integers index into.
Each tokenizer assigns its own integers to its own vocabulary. The same id in two different tokenizers points to different rows of two different embedding matrices, with no relationship between them.
Stuck? Peek. Progressively more specific. Each peek narrows the answer further.
A bigger vocabulary buys you something on one axis and costs you something on another. Think about what changes about the input sequence and what changes about the model parameters.
Two large parameter matrices in the model have vocab_size as one dimension. Making the vocabulary bigger makes those matrices bigger; making it smaller makes input sequences longer.
Stuck? Peek. Progressively more specific. Each peek narrows the answer further.
When you call AutoTokenizer.from_pretrained, the loader reads more than one file. One of them is the vocabulary; the other is a configuration sidecar. Think about what configuration the loader needs that is not part of the vocabulary itself.
The vocabulary file is large because it holds all the tokens and merges. The config file is small because it only holds settings. The two together fully describe the tokenizer's behavior.
Use tiktoken.get_encoding('cl100k_base') and len(encoded), since Claude uses the same BPE family as GPT-4
Call Anthropic's messages.count_tokens endpoint with the formatted message list and read input_tokens from the response
Estimate len(text) / 4 as a rule of thumb, since most production token counts are within 10 percent of this heuristic
Load the deprecated Claude 2 tokenizer from the old @anthropic-ai/tokenizer package and use it for all Claude versions including Sonnet 4.6
Stuck? Peek. Progressively more specific. Each peek narrows the answer further.
Anthropic does not ship a public tokenizer library for Claude. What does that constraint imply about which token-counting methods can possibly be accurate?
Token counts are the bill. If your estimate is off by 20 percent, your cost model is wrong by 20 percent and your context-budget logic truncates at the wrong byte. Of the four options, which one gives counts that match Anthropic's billing most closely?
Stuck? Peek. Progressively more specific. Each peek narrows the answer further.
Tokens carry two kinds of meaning: content meaning (this is a word fragment) and structural meaning (this is a boundary). Think about which ids represent the second kind.
Suppose user input flows directly into the tokenizer without sanitization. What could go wrong if that input contains a string that looks like one of these reserved markers? The answer is the prompt injection class of bugs.
Name four common special tokens in modern LLMs and describe what goes wrong if BOS or EOS is missing.
Stuck? Peek. Progressively more specific. Each peek narrows the answer further.
Think about the sequence lifecycle: something marks the start, something marks the end, something fills empty positions in a batch. What is the concrete failure when the 'end' marker is absent during generation?
Consider the attention mechanism's behavior with PAD tokens. If PAD tokens are included in the attention computation without masking, the model attends to padding positions. What does that do to the weighted sum over values: does it add noise, zero out values, or something else?