Predict how many tokens the rainbow flag emoji 🏳️‍🌈 uses in a byte level BPE tokenizer with no emoji in training data.

The surprising part of this question is that one emoji you see as a single picture is, to the tokenizer, a long string of bytes. A **byte-level BPE** tokenizer starts with the 256 byte values and learns shortcuts, called merges, only for patterns it saw often in training. The example tokenizer never saw emoji, so it learned no shortcuts for them. Without a shortcut, the tokenizer has to spell the emoji out byte by byte. The rainbow flag 🏳️‍🌈 is not one character. It is four Unicode pieces, a white flag, a variation selector, an invisible Zero-Width Joiner (ZWJ), and a rainbow, glued together. In UTF-8 those add up to about 14 bytes. So the prediction is about 14 tokens, roughly one per byte. Compare 'hello', which is 5 bytes and usually merges into 1 or 2 tokens. The emoji costs roughly 7 to 14 times more, even though it looks smaller on screen. That gap is the byte-fallback cost.

About **14 tokens**, and the reasoning matters more than the number. Byte-level BPE seeds its vocabulary with the 256 byte values, then learns merges only for frequent training patterns. The given tokenizer saw no emoji, so no merge rule applies. With zero applicable merges, the floor is one token per byte: $$\#\,\text{tokens} = \#\,\text{bytes} \quad (\text{no merges apply})$$ Now count the bytes. The rainbow flag is a ZWJ sequence of four codepoints. The pieces are the white flag (4 bytes), a variation selector (3 bytes), the Zero-Width Joiner U+200D (3 bytes), and the rainbow (4 bytes). That sums to roughly 14 bytes, hence roughly 14 tokens. The insight an interviewer is listening for is that visual size is irrelevant. The emoji renders as one compact glyph but is structurally a multi-codepoint sequence, and the tokenizer charges per byte, not per glyph. A 5-byte word like 'hello' merges down to 1-2 tokens, while this single emoji costs around 14. That is the byte-fallback tax for any symbol absent from training data.