Flashcard

An LLM never sees raw text. So what does it process?

Before the tokenizer even starts splitting, normalization runs. What does it do and why?

Pre-tokenization splits text before BPE merges. Why is that boundary important?

A tokenizer has a vocabulary. What is it, and why does its size matter for the model?

Token 1234 means "the" in one model and garbage in another. Why?

Your team debates 32K vs 128K vs 256K vocab. What is the core tradeoff they should frame?

You open a HuggingFace model repo and see tokenizer_config.json. What is in it and what is NOT?

BOS, EOS, PAD: why do tokenizers reserve these ids instead of using normal text?

What are BOS, EOS, PAD, and role marker special tokens, and what happens when each is missing or duplicated?

You pad a batch to equal length. How does the model know to ignore the filler tokens?

Which of bytes, characters, or tokens does an LLM API charge you for, and how do the three relate?

Your teammate mentions BPE at a code review. What are they talking about?

Why use subword tokens, and what is wrong with word level or character level?

BPE learned 50,000 merge rules during training. What is a merge rule, and where does it live at inference?

BPE grows the vocabulary by adding merges. Unigram does the opposite. Explain the difference.

A tokenizer is byte level. What does that buy you over a character level one?

How does byte level BPE guarantee no unknown tokens, and what is the cost of that guarantee?

Older tokenizers had an OOV problem. Why did byte level BPE kill it?

You hear 'fertility is 5 for Hindi on cl100k_base'. What does that number mean and is it good?

A chat template wraps every turn with BOS and EOS tokens. Why does the model need them?

Someone confuses 'sequence length' with 'context window'. How do you correct them?

Your input is 9,000 tokens but max_length is 8,192. What are your truncation options?

You change one word in the system prompt and prompt caching breaks. Why?

What are BPE merge residues (LiteToken) and what do they imply for vocabulary efficiency?

What constraint does BoundlessBPE relax, and what improvement does it yield?