Glossary · Foundations
BPE (Byte-Pair Encoding)
Also known as: Byte-Pair Encoding, Byte-level BPE
Tokenizer algorithm that learns a vocabulary by greedily merging the most frequent character pairs.
A subword tokenization algorithm that starts with characters and iteratively merges the most frequent adjacent pair into a new token. Used by GPT-2/3/4, LLaMA, and most modern LLMs. 'Byte-level BPE' operates on UTF-8 bytes so any text can be encoded losslessly.
In practice
How GPT-4, LLaMA, and most modern LLMs tokenize. Interviews probe why "hello world" is 2 tokens but the same word in Hindi is 12.
How it compares
BPE is one specific tokenization algorithm; tokenization is the general process.