Multilingual

You hear 'fertility is 2.5 for Hindi on cl100k_base'. What does that number mean and is it good?

You are building a 7B model for 25 languages including Hindi, Arabic, and Swahili. Design a fair tokenizer and name what it costs.

The same sentence in English, Spanish, Hindi, and Japanese. Predict the token ratio on o200k_base for each.

How does vocabulary size affect embedding table memory footprint at bfloat16 precision, and what are the tradeoffs of very small vs. very large vocabularies?

What is the 'token tax' and how does it create downstream accuracy disparities across languages?

What primarily causes the 'token tax' for low-resource languages in BPE tokenizers?

What is STRR and how does it reveal poor tokenizer coverage that fertility alone misses?

Why does SentencePiece prepend a special space symbol (▁) to tokens, and what would break without it?

Which tokenizer can learn merges that span whitespace boundaries by design?

Spot the failure mode: a tokenizer trained with NFC normalization is deployed with an NFKC preprocessing pipeline.

Define 'fertility' as a tokenizer metric and explain why high fertility for a language is both a fairness and a model performance problem.

According to tokenizer fairness research, if a language's fertility doubles, what typically happens to downstream model accuracy?

Estimate the token cost difference for processing 1M Spanish/French product descriptions vs. an English-only baseline, and describe correct budget planning.

What most inflates token count for European languages compared to English in a BPE tokenizer trained on English-dominant data?

How would you build an accurate token budget estimator for a multilingual RAG application?

What does the '200k' in o200k_base signify, and what was improved compared to cl100k_base?

How does tokenizer fertility bias affect cost and context window budget for non-English API users?

Predict the approximate token count difference between 'Hello world' and its Arabic equivalent 'مرحبا بالعالم' in a GPT-4 tokenizer.