How does WordPiece decide which pair to merge, compared to BPE?

Both **WordPiece** and BPE build a vocabulary the same way at a high level: start from characters, repeatedly merge an adjacent pair into a new symbol, and stop at a target size. The difference is which pair they pick at each step. BPE is blunt. It chooses the pair that appears the most times in the corpus. If 'th' shows up more than anything else, 'th' merges first. WordPiece picks by a likelihood score instead. For a pair of pieces A and B, it computes how often they appear together divided by how often each appears on its own: $$\text{score}(A,B) = \frac{\text{freq}(AB)}{\text{freq}(A)\,\cdot\,\text{freq}(B)}$$ This rewards pairs that truly belong together, not pairs that are just individually common. A piece like 'e' is everywhere, so any pair containing it has a big denominator and a small score. A genuine fragment like 'qu' has pieces that co-occur far more than chance, so it scores high and merges. The payoff is a vocabulary that lines up with real word structure, which is why BERT uses it.

Both algorithms grow a vocabulary by greedily merging adjacent pairs from a character start, but the **merge criterion** differs. BPE picks the pair with the highest raw count. WordPiece picks the pair that most increases the training corpus likelihood, which reduces to a normalized score: $$\text{score}(A,B) = \frac{\text{freq}(AB)}{\text{freq}(A)\,\cdot\,\text{freq}(B)}$$ The denominator is the key. It divides out how common each piece is on its own, so the score measures co-dependence, not co-occurrence. A pair like 'e' plus a space is hugely frequent, but its pieces are everywhere, so the ratio stays small. A pair like 's' plus 'h' forming 'sh' co-occurs far above chance, so it scores high. The practical effect is that WordPiece tends to land on linguistically meaningful subwords, which is why BERT and its descendants use it. The trade-off shows up at encode time: WordPiece does greedy longest-match and can emit `[UNK]` if no vocabulary entry covers a substring, whereas byte-level BPE never does. So WordPiece buys cleaner morphology at the cost of possible unknown tokens.