Bpe

Predict how many tokens the rainbow flag emoji 🏳️‍🌈 uses in a byte level BPE tokenizer with no emoji in training data.

How does byte-level BPE guarantee no unknown tokens, and what is the cost of that guarantee?

How does weighting BPE merges by document frequency instead of raw token frequency change the resulting vocabulary?

Put the BPE training steps in the correct order, from raw corpus to final vocabulary.

Spot the error: 'BPE finds the globally optimal vocabulary by iteratively merging the most frequent pair.'

BPE-dropout is applied at which stage(s) of the LLM lifecycle?

What constraint does BoundlessBPE relax, and what improvement does it yield?