Tokx Vocabulary

A tokenizer has a 'vocabulary'. What is it, and why does its size matter for the model?

An LLM never sees raw text. So what does it actually process?

BOS, EOS, PAD: why do tokenizers reserve these ids instead of using normal text?

Pre-tokenization splits text before BPE merges. Why is that boundary important?

Older tokenizers had an OOV problem. Why did byte-level BPE kill it?

Before the tokenizer even starts splitting, normalization runs. What does it do and why?

You hear 'fertility is 2.5 for Hindi on cl100k_base'. What does that number mean and is it good?

A tokenizer is 'byte-level'. What does that buy you over a character-level one?

Your teammate mentions BPE at a code review. What are they talking about?