Your teammate mentions BPE at a code review. What are they talking about?

Flashcard·Easy·4.0 · 0·~30s·Asked atMckinsey Meta OpenAI

Attempt it

BPE is a tokenizer training algorithm that starts from individual bytes or characters and repeatedly merges the most frequent adjacent pair into a new symbol until it reaches a target vocabulary size. The result is a fixed vocabulary plus a deterministic encoder, and byte-level BPE is the dominant sub-word tokenizer in 2026: it underlies tiktoken (cl100k_base, o200k_base), GPT-2 through GPT-5.5, Llama 3+, Mistral, and DeepSeek V4.

Your answer

**BPE (Byte Pair Encoding)** is a tokenizer training algorithm that starts from individual bytes or characters and repeatedly merges the most frequent adjacent pair into a new symbol. It keeps merging until it reaches a target vocabulary size, then stops. The output of training is two things: a fixed vocabulary (the list of all symbols, including the original alphabet plus every merged symbol) and an ordered merge table (the rules learned in order). Together they form a deterministic encoder. At inference time, BPE applies those merges to any input string in the same order they were learned. The result is a sequence of token ids. Because the same merges are applied in the same order every time, the encoder is deterministic: the same string always produces the same token sequence. Byte-level BPE, the modern variant, starts from the 256 possible byte values instead of characters. This guarantees that any UTF-8 input can be encoded losslessly with no out of vocabulary case. In 2026, byte-level BPE is the dominant tokenizer family: tiktoken (cl100k_base for GPT-4 and GPT-3.5-turbo, o200k_base for GPT-4o, o1, o3, GPT-5, GPT-5.5), the Llama 3+ tokenizer, Mistral, and DeepSeek V4 all use it.

**BPE (Byte Pair Encoding)** is a sub-word tokenizer training algorithm originally adapted from a 1994 data compression technique. It starts from a minimal alphabet (raw bytes or characters), counts the frequency of every adjacent symbol pair in the training corpus, merges the most frequent pair into a new symbol, and repeats. Training stops when the vocabulary reaches a target size (commonly 32K to 200K for production LLMs). The training phase produces two artifacts: an ordered merge table and a final vocabulary. The merge table lists every learned merge in the order it was created; the vocabulary maps each final symbol to an integer id. Together they fully determine the tokenizer. At inference time, BPE applies the merges to any new string greedily in learned order. Each input is first split by a pre-tokenization regex into coarse chunks, then within each chunk the merge rules are applied repeatedly until no more apply. The result is a sequence of token ids. Byte-level BPE, introduced by GPT-2, starts from the 256 possible byte values instead of Unicode characters. This guarantees losslessness: any UTF-8 byte sequence can be expressed in the vocabulary, so there is no out of vocabulary case by construction. Where it shows up in 2026. OpenAI's tiktoken (cl100k_base, o200k_base) is byte-level BPE. The Llama 3 and Llama 4 tokenizers are tiktoken-style byte-level BPE with a 128K vocabulary. Mistral uses its own SentencePiece-based BPE with a 32K vocabulary. DeepSeek V4 uses byte-level BPE. BPE is the default in modern LLM tokenization.

TL;DR

BPE is a sub-word tokenizer algorithm: start from bytes or characters, merge the most frequent adjacent pair, repeat until the vocabulary hits a target size. Output is a vocabulary plus a deterministic encoder.

Memory aid

Easy to grasp

Imagine you are inventing a shorthand for a long book. You start by writing every single letter separately. Then you notice that 't' followed by 'h' shows up everywhere, so you invent a single new symbol that means 'th' and replace every occurrence. Now you scan again and find that 'th' followed by 'e' is the next most common pair, so you invent a symbol for 'the' and replace those. You keep doing this until you have invented as many new symbols as you can afford. Frequent words end up with their own short symbols; rare words stay spelled out from smaller pieces. That is exactly what BPE does, except the alphabet starts from raw bytes instead of letters, and the merging happens automatically over billions of words of training text. The result is a vocabulary the language model uses to read every prompt for the rest of its life.

Key concepts

Byte Pair Encoding Merge Rule Subword Unit tiktoken Vocabulary

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

BPE (Byte Pair Encoding) is the workhorse tokenizer of the 2020s LLM era. The original algorithm dates to a 1994 data compression paper; Sennrich et al. adapted it for neural machine translation in 2015; GPT-2 in 2019 introduced the byte-level variant that now dominates. By 2026 the OpenAI, Meta, Mistral, and DeepSeek model families all ship with byte-level BPE tokenizers.

This deep dive walks the training loop, separates training from inference, explains the byte-level twist, and places BPE against its main alternatives. The goal is for the reader to be able to recognize a BPE tokenizer in any context and reason about its behavior on new strings.

The training loop, end to end

Training runs once, offline, on a fixed corpus. The procedure has four steps that repeat.

First, initialize the vocabulary with the base alphabet. For classical BPE this is the set of unique characters in the corpus. For byte-level BPE this is the 256 possible byte values, regardless of the corpus.

Second, represent the corpus as a sequence of base-alphabet symbols and count the frequency of every adjacent pair. Frequency is summed across the entire corpus.

Third, pick the most frequent pair. Add a new symbol that represents that pair to the vocabulary and record the merge rule (left, right, merged).

Fourth, scan the corpus and replace every occurrence of the chosen pair with the new symbol, then return to step two with the updated corpus and updated vocabulary.

The loop stops when the vocabulary reaches the target size (commonly 32K, 50K, 100K, or 200K for production LLMs). The output of training is the final vocabulary and the ordered list of merges. These two artifacts are the entire tokenizer; together they define a deterministic function from text to token ids.

A worked example: the corpus 'low low low lower lowest' with character-level BPE would start with the alphabet {l, o, w, e, r, s, t}. The most frequent pair is 'l, o' (5 occurrences). Merge to 'lo'. Next most frequent pair is 'lo, w' (5 occurrences). Merge to 'low'. After two merges the vocabulary already contains the word 'low' as a single symbol. Continuing the loop builds up 'lower' and 'lowest' as derived forms. Real training runs the same loop on billions of words, producing tens of thousands of merges.

Inference: applying the learned merges

The byte-level twist

Where BPE sits in the 2026 tokenizer landscape

Known limits and active variants

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

OpenAI's tiktoken library implements byte-level BPE; cl100k_base is the encoding for GPT-4 and GPT-3.5-turbo, and o200k_base is the encoding for GPT-4o, o1, o3, GPT-5, and GPT-5.5.
Meta's Llama 3 and Llama 4 tokenizers are tiktoken-style byte-level BPE with a 128K vocabulary, a deliberate switch from the 32K SentencePiece tokenizer used in Llama 2.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow is byte-level BPE different from classical character-level BPE?

Byte-level BPE replaces the character alphabet with the 256 possible UTF-8 byte values. This guarantees losslessness on any input and removes the OOV problem entirely. Walk through how a non-Latin script or emoji decomposes into byte-level tokens that the model has seen during pretraining.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Confusing the training phase that builds the merge table with the inference phase that applies it. Training picks the merges once; inference applies them deterministically to every new string.

60 second bullets to scan on the way to the call.

Define BPE as a sub-word tokenizer training algorithm in one sentence.
Describe the training loop: start from base alphabet, count pairs, merge the most frequent, repeat.

Primary sources. Browse if you want the original framing.