Fairness

You are building a 7B model for 25 languages including Hindi, Arabic, and Swahili. Design a fair tokenizer and name what it costs.

What is the 'token tax' and how does it create downstream accuracy disparities across languages?

What primarily causes the 'token tax' for low resource languages in BPE tokenizers?

What is STRR and how does it reveal poor tokenizer coverage that fertility alone misses?

What is STRR and how does it complement fertility as a tokenizer fairness metric?

Define 'fertility' as a tokenizer metric and explain why high fertility for a language is both a fairness and a model performance problem.

According to tokenizer fairness research, if a language's fertility doubles, what typically happens to downstream model accuracy?