Subword

An LLM never sees raw text. So what does it actually process?

Your teammate mentions BPE at a code review. What are they talking about?

Character, word, and subword tokenization each have a dominant failure mode. Match them.

BPE and WordPiece both merge subwords. What is the one thing they disagree on?

BPE grows the vocabulary by adding merges. Unigram does the opposite. Explain the difference.

Explain why word-level and character-level tokenization both fail for modern LLMs, and how subword tokenization resolves each failure mode.

Why do modern LLMs use subword tokenization instead of word-level or character-level approaches?