AI Researcher

What is the 'token tax' and how does it create downstream accuracy disparities across languages?

What primarily causes the 'token tax' for low-resource languages in BPE tokenizers?

Explain why word-level and character-level tokenization both fail for modern LLMs, and how subword tokenization resolves each failure mode.

What is STRR and how does it reveal poor tokenizer coverage that fertility alone misses?

What is STRR and how does it complement fertility as a tokenizer fairness metric?

What concretely breaks when you try to load a pretrained model checkpoint with a different tokenizer vocabulary?

Why is it not possible to simply swap a pretrained LLM's tokenizer for one with a larger vocabulary?

Why does SentencePiece prepend a special space symbol (▁) to tokens, and what would break without it?

Which tokenizer can learn merges that span whitespace boundaries by design?

What happens to emoji ZWJ sequences under cl100k_base vs. o200k_base pre-tokenization?

What is the primary purpose of regex-based pre-tokenization in cl100k_base, and what constraint does it impose on BPE?

Why do LLMs frequently fail at arithmetic involving multi-digit numbers? Trace the failure to tokenization.

Predict how cl100k_base tokenizes '1024', '2048', and '100000', and explain how inconsistent splitting causes LLM arithmetic failures.

Which Unicode normalization does BERT's tokenizer apply, and how does it differ from NFC?

Approximately what fraction of BPE vocabulary tokens does LiteToken identify as 'merge residues'?

What are BPE merge residues (LiteToken) and what do they imply for vocabulary efficiency?

Define 'fertility' as a tokenizer metric and explain why high fertility for a language is both a fairness and a model performance problem.

According to tokenizer fairness research, if a language's fertility doubles, what typically happens to downstream model accuracy?

What does the '200k' in o200k_base signify, and what was improved compared to cl100k_base?

Predict the approximate token count difference between 'Hello world' and its Arabic equivalent 'مرحبا بالعالم' in a GPT-4 tokenizer.

How does weighting BPE merges by document frequency instead of raw token frequency change the resulting vocabulary?

Put the BPE training steps in the correct order, from raw corpus to final vocabulary.

Spot the error: 'BPE finds the globally optimal vocabulary by iteratively merging the most frequent pair.'

BPE-dropout is applied at which stage(s) of the LLM lifecycle?

What interpretability and downstream task concerns arise from allowing cross-word boundary merges in BoundlessBPE?