Embedding Matrix

A tokenizer has a 'vocabulary'. What is it, and why does its size matter for the model?

Your team debates 32K vs 128K vs 256K vocab. What is the core tradeoff they should frame?

Token 1234 means 'the' in one model and garbage in another. Why?

How does vocabulary size affect embedding table memory footprint at bfloat16 precision, and what are the tradeoffs of very small vs. very large vocabularies?

Which part of a transformer model's parameter count scales directly and linearly with vocabulary size?

What would break if you tokenized a Llama prompt using tiktoken's cl100k_base instead of the Llama tokenizer?

What concretely breaks when you try to load a pretrained model checkpoint with a different tokenizer vocabulary?

Why is it not possible to simply swap a pretrained LLM's tokenizer for one with a larger vocabulary?