Tokx Vocabulary

A tokenizer has a 'vocabulary'. What is it, and why does its size matter for the model?

An LLM never sees raw text. So what does it actually process?

BOS, EOS, PAD: why do tokenizers reserve these ids instead of using normal text?

Pre-tokenization splits text before BPE merges. Why is that boundary important?

Older tokenizers had an OOV problem. Why did byte level BPE kill it?

Before the tokenizer even starts splitting, normalization runs. What does it do and why?

You hear 'fertility is 2.5 for Hindi on cl100k_base'. What does that number mean and is it good?

A tokenizer is 'byte level'. What does that buy you over a character level one?

Your teammate mentions BPE at a code review. What are they talking about?