Tokx Pattern Glossary

Your team debates 32K vs 128K vs 256K vocab. What is the core tradeoff they should frame?

You open a HuggingFace model repo and see tokenizer_config.json. What is in it and what is NOT?

Token 1234 means 'the' in one model and garbage in another. Why?

BPE learned 50,000 merge rules during training. What is a merge rule, and where does it live at inference?

You need to count tokens locally. When do you reach for tiktoken versus SentencePiece?

Character, word, and subword tokenization each have a dominant failure mode. Match them.

BPE and WordPiece both merge subwords. What is the one thing they disagree on?

BPE grows the vocabulary by adding merges. Unigram does the opposite. Explain the difference.