Zenaique
Glossary · Foundations

Tokenization

Also known as: BPE, WordPiece, SentencePiece

Splitting text into sub-word units so a model can consume it as a sequence of IDs.

The process of splitting raw text into sub-word units (tokens) that serve as the model's vocabulary. Common algorithms include BPE, WordPiece, and SentencePiece. Token count directly affects cost and context window usage.

In practice

Token count drives cost and context-window math. Expect questions on BPE merges, why "strawberry" gets split weirdly, and the difference between byte-level and character-level BPE.

Comparisons that include Tokenization

Semantic Search vs Keyword SearchUse semantic search when users phrase things their own way; keep keyword search when exact terms like codes and IDs must match. Most production systems run both.

Related topics

Tokenization

95 questions

Practice questions

Questions that mention this term

Related terms

BPE (Byte-Pair Encoding)

Tokenizer algorithm that learns a vocabulary by greedily merging the most frequent character pairs.