Concepts · Foundations
Tokenization
Also known as: BPE, WordPiece, SentencePiece
Splitting text into sub-word units so a model can consume it as a sequence of IDs.
The process of splitting raw text into sub-word units (tokens) that serve as the model's vocabulary. Common algorithms include BPE, WordPiece, and SentencePiece. Token count directly affects cost and context window usage.
In practice
Token count drives cost and context-window math. Expect questions on BPE merges, why "strawberry" gets split weirdly, and the difference between byte-level and character-level BPE.
Comparisons that include Tokenization
Related topics
Practice questions
Questions that mention this term
- A chatbot team reports TTFT P99 jumped from 300ms to 1.4s overnight. Which root cause is most likely?
- Pair each decoding strategy with its determinism and diversity profile.
- Chain-of-Thought is widely cited as 'always improves accuracy'. For which of these tasks would adding CoT prompting most reliably help?
- Can a smaller model beat a larger one at equal compute?
- Identify the concrete lock in costs a framework imposes on a production codebase (select all that apply)
- When should you use token level F1 vs exact match in a QA evaluation?