Fairness

You are building a 7B model for 25 languages including Hindi, Arabic, and Swahili. Design a fair tokenizer and name what it costs.

What is the 'token tax' and how does it create downstream accuracy disparities across languages?

What primarily causes the 'token tax' for low-resource languages in BPE tokenizers?

What is STRR and how does it reveal poor tokenizer coverage that fertility alone misses?

What is STRR and how does it complement fertility as a tokenizer fairness metric?

Define 'fertility' as a tokenizer metric and explain why high fertility for a language is both a fairness and a model performance problem.

According to tokenizer fairness research, if a language's fertility doubles, what typically happens to downstream model accuracy?