The per question revision bullets, not the topic cheatsheets.
60 second bullets to scan on the way to the call.
State in one sentence what a token is at the byte level.
Explain why token count is not the same as character count or word count.
Character level tokenization
Word level tokenization
Subword tokenization (BPE, WordPiece, Unigram LM)
Byte level subword tokenization (byte level BPE)
Closed vocabulary; any unseen word becomes [UNK] and information is lost
No native failure mode at this resolution; this is the Pareto compromise modern LLMs use
Latin script bias: non-Latin scripts often need many bytes per character, inflating token counts and costs
Sequence length explosion that blows up attention cost and shrinks effective context
60 second bullets to scan on the way to the call.
Name the dominant failure mode of character-level tokenization.
Name the dominant failure mode of word-level tokenization.
Word level tokenizers fail on new words not seen during training, while character level tokenizers produce sequences far too long for attention to process efficiently.
Subword tokenizers always produce shorter sequences than word level tokenizers for the same input text.
Character level tokenizers require a larger vocabulary than subword tokenizers, making them memory inefficient.
Word level tokenizers use too much memory because each word needs its own embedding vector.
60 second bullets to scan on the way to the call.
The word-level coverage failure and the UNK token
The character-level sequence-length explosion
Using a standard GPT style BPE tokenizer on the string:
'hello world'
Approximate token count?
60 second bullets to scan on the way to the call.
Why common English words become single tokens in modern BPE
How GPT-style pre-tokenization handles the leading space
SentencePiece applies NFKC normalization, which additionally decomposes compatibility characters (half width katakana, ligatures, circled numbers) unlike NFC which only handles canonical equivalences.
SentencePiece applies NFC normalization, which is identical to NFKC for all practical purposes since both produce the same byte sequences for standard text.
SentencePiece applies no Unicode normalization and passes raw bytes directly to its segmentation model.
SentencePiece applies NFD normalization, which decomposes composed characters into their base + combining mark form before tokenization.
60 second bullets to scan on the way to the call.
The four Unicode normalization forms and which fold compatibility characters
What the K in NFKC and NFKD stands for
60 second bullets to scan on the way to the call.
Define tokenizer normalization in one sentence.
Name the four Unicode normalization forms (NFC, NFD, NFKC, NFKD).
60 second bullets to scan on the way to the call.
Define pre-tokenization in one sentence.
Identify where it sits in the tokenizer pipeline (after normalization, before merging).
60 second bullets to scan on the way to the call.
Define the vocabulary as the numbered, fixed set of token ids the tokenizer can emit.
Identify the two model components whose shape depends on vocab_size.
60 second bullets to scan on the way to the call.
Define a token id as an integer index into a specific tokenizer's vocabulary.
Name the two model components token ids interact with (embedding matrix at input, lm_head at output).
60 second bullets to scan on the way to the call.
State the headline tradeoff in one sentence: sequence compression versus parameter cost.
Name the two large matrices that scale with vocab_size (embedding and lm_head).
60 second bullets to scan on the way to the call.
Define tokenizer_config.json as the HuggingFace sidecar for non-vocabulary settings.
Name at least four entries that typically appear in the file.
Use tiktoken.get_encoding('cl100k_base') and len(encoded), since Claude uses the same BPE family as GPT-4
Call Anthropic's messages.count_tokens endpoint with the formatted message list and read input_tokens from the response
Estimate len(text) / 4 as a rule of thumb, since most production token counts are within 10 percent of this heuristic
Load the deprecated Claude 2 tokenizer from the old @anthropic-ai/tokenizer package and use it for all Claude versions including Sonnet 4.6
60 second bullets to scan on the way to the call.
Why Anthropic does not let you tokenize locally for Claude.
What the messages.count_tokens endpoint returns and what fields it accepts.
60 second bullets to scan on the way to the call.
Define a special token as a reserved vocabulary entry carrying structural rather than content meaning.
Name the standard categories: sentinels (BOS, EOS), PAD, role markers, tool-use markers.
Name four common special tokens in modern LLMs and describe what goes wrong if BOS or EOS is missing.
60 second bullets to scan on the way to the call.
The four special token families and what each one is for
Why a missing EOS lets generation run to the length cap