Your teammate mentions BPE at a code review. What are they talking about?

Flashcard·Easy·4.0 · 0·~30s·Relevant atOpenAI

Attempt it

BPE is a tokenizer training algorithm that starts from individual bytes or characters and repeatedly merges the most frequent adjacent pair into a new symbol until it reaches a target vocabulary size. The result is a fixed vocabulary plus a deterministic encoder, and byte-level BPE is the dominant sub-word tokenizer in 2026: it underlies tiktoken (cl100k_base, o200k_base), GPT-2 through GPT-5.5, Llama 3+, Mistral, and DeepSeek V4.

Your answer

**BPE (Byte Pair Encoding)** is a tokenizer training algorithm that starts from individual bytes or characters and repeatedly merges the most frequent adjacent pair into a new symbol. It keeps merging until it reaches a target vocabulary size, then stops. The output of training is two things: a fixed vocabulary (the list of all symbols, including the original alphabet plus every merged symbol) and an ordered merge table (the rules learned in order). Together they form a deterministic encoder. At inference time, BPE applies those merges to any input string in the same order they were learned. The result is a sequence of token ids. Because the same merges are applied in the same order every time, the encoder is deterministic: the same string always produces the same token sequence. Byte-level BPE, the modern variant, starts from the 256 possible byte values instead of characters. This guarantees that any UTF-8 input can be encoded losslessly with no out of vocabulary case. In 2026, byte-level BPE is the dominant tokenizer family: tiktoken (cl100k_base for GPT-4 and GPT-3.5-turbo, o200k_base for GPT-4o, o1, o3, GPT-5, GPT-5.5), the Llama 3+ tokenizer, Mistral, and DeepSeek V4 all use it.

**BPE (Byte Pair Encoding)** is a sub-word tokenizer training algorithm originally adapted from a 1994 data compression technique. It starts from a minimal alphabet (raw bytes or characters), counts the frequency of every adjacent symbol pair in the training corpus, merges the most frequent pair into a new symbol, and repeats. Training stops when the vocabulary reaches a target size (commonly 32K to 200K for production LLMs). The training phase produces two artifacts: an ordered merge table and a final vocabulary. The merge table lists every learned merge in the order it was created; the vocabulary maps each final symbol to an integer id. Together they fully determine the tokenizer. At inference time, BPE applies the merges to any new string greedily in learned order. Each input is first split by a pre-tokenization regex into coarse chunks, then within each chunk the merge rules are applied repeatedly until no more apply. The result is a sequence of token ids. Byte-level BPE, introduced by GPT-2, starts from the 256 possible byte values instead of Unicode characters. This guarantees losslessness: any UTF-8 byte sequence can be expressed in the vocabulary, so there is no out of vocabulary case by construction. Where it shows up in 2026. OpenAI's tiktoken (cl100k_base, o200k_base) is byte-level BPE. The Llama 3 and Llama 4 tokenizers are tiktoken-style byte-level BPE with a 128K vocabulary. Mistral uses its own SentencePiece-based BPE with a 32K vocabulary. DeepSeek V4 uses byte-level BPE. BPE is the default in modern LLM tokenization.

Quick reference

TL;DR

BPE is a sub-word tokenizer algorithm: start from bytes or characters, merge the most frequent adjacent pair, repeat until the vocabulary hits a target size. Output is a vocabulary plus a deterministic encoder.

Memory aid

Easy to grasp

Imagine you are inventing a shorthand for a long book. You start by writing every single letter separately. Then you notice that 't' followed by 'h' shows up everywhere, so you invent a single new symbol that means 'th' and replace every occurrence. Now you scan again and find that 'th' followed by 'e' is the next most common pair, so you invent a symbol for 'the' and replace those. You keep doing this until you have invented as many new symbols as you can afford. Frequent words end up with their own short symbols; rare words stay spelled out from smaller pieces. That is exactly what BPE does, except the alphabet starts from raw bytes instead of letters, and the merging happens automatically over billions of words of training text. The result is a vocabulary the language model uses to read every prompt for the rest of its life.

Key concepts

Byte Pair Encoding Merge Rule Subword Unit tiktoken Vocabulary

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: state the algorithm in one breath, walk the training loop on a tiny example, separate training from inference, explain the byte-level twist and what it buys, and name three 2026 model families that ship with byte-level BPE.

Real-world examples

Real products, models, and research that use this idea.

OpenAI's tiktoken library implements byte-level BPE; cl100k_base is the encoding for GPT-4 and GPT-3.5-turbo, and o200k_base is the encoding for GPT-4o, o1, o3, GPT-5, and GPT-5.5.
Meta's Llama 3 and Llama 4 tokenizers are tiktoken-style byte-level BPE with a 128K vocabulary, a deliberate switch from the 32K SentencePiece tokenizer used in Llama 2.
Mistral models (Mistral Large 3, Mixtral) use a SentencePiece-based BPE tokenizer with a 32K vocabulary.
The Hugging Face tokenizers library (Rust core, Python bindings) provides production-grade BPE training and inference and is the standard for custom-vocabulary work.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow is byte-level BPE different from classical character-level BPE?

QHow does BPE differ from WordPiece in selecting which pair to merge next?

QWhat does BPE-Dropout do and why?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Confusing the training phase that builds the merge table with the inference phase that applies it. Training picks the merges once; inference applies them deterministically to every new string.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.