BPE learned 50,000 merge rules during training. What is a merge rule, and where does it live at inference?

Flashcard·Easy·4.0 · 0·~30s·Asked atCoinbase Fireworks Ai Lakera·Relevant atOpenAI

Attempt it

A merge rule is one entry in the ordered list BPE learned during training: 't' + 'h' becomes 'th', then 'th' + 'e' becomes 'the', and so on. At encoding time the tokenizer applies the merges greedily in their learned order to produce the final token sequence. Merges live in the tokenizer artifact: a merges.txt file for GPT-2-style tokenizers, embedded inside tokenizer.json for HuggingFace fast tokenizers, and embedded in the binary encoder for tiktoken.

Your answer

**A merge rule** is one entry in the ordered list that BPE produced during training. Each entry says 'two specific pieces combine into one new piece'. The list might start with 't' + 'h' becomes 'th', then continue with 'in' + 'g' becomes 'ing', then 'th' + 'e' becomes 'the', and so on, for tens of thousands of entries. At encoding time, the tokenizer applies these merges to any input string. It splits the input into the base alphabet (bytes or characters), then walks through its merge list in learned order and applies every applicable merge greedily. Once no more merges apply, the resulting sequence of pieces is the tokenization. The order is critical because earlier merges create the pieces that later merges combine. Where the list lives depends on the library: - **GPT-2-style tokenizers** store merges in a plain text file called merges.txt, one merge per line. - **HuggingFace fast tokenizers** embed the merges (along with vocabulary and pre-tokenization rules) inside a single tokenizer.json file. - **tiktoken** packs the merges into a compact binary encoder that ships with the OpenAI Python package; you do not see them as a separate file. - **SentencePiece BPE** packs them into the .model protobuf file alongside the vocabulary. In 2026 the common format is tokenizer.json for non-OpenAI models and the tiktoken binary for OpenAI models.

**A merge rule** is one entry in the ordered list of pair-token combinations that BPE produced during training. The list is the core artifact of BPE; everything else (the vocabulary itself) can be derived from it given a base alphabet. During training, BPE starts from a base alphabet (bytes for byte-level BPE, characters for classical BPE) and iteratively picks the most frequent adjacent pair across the corpus, merges every occurrence into a new piece, and records the merge as a rule. The output of training is a vocabulary (the base alphabet plus every merged piece) and an ordered merge list. At encoding time, the tokenizer applies the merges greedily in the order they were learned. The pseudocode is: 1. Split the input string into the base alphabet (bytes or characters). 2. Walk the merge list in order. For each rule '(A, B) becomes AB', scan the current piece sequence and merge every adjacent occurrence of A followed by B into AB. 3. Continue until the merge list is exhausted. The order matters because earlier merges create the pieces that later merges combine. Encoding is not a search for the optimal segmentation; it is a deterministic replay of the learned merges. The storage format varies by library: - **GPT-2-style and early HuggingFace tokenizers**: a plain-text merges.txt file with one merge per line (space-separated pair) plus a separate vocab.json mapping each token to its id. - **HuggingFace fast tokenizers (2026 standard)**: a single tokenizer.json file that encodes the pre-tokenizer, the model (including merges and vocab), the post-processor, and the decoder. - **tiktoken**: the merges and vocabulary are packed into a compact binary blob that the encoder reads at load time. You do not see merges.txt; the list is in the binary. - **SentencePiece BPE**: merges and vocabulary live inside the .model protobuf file. The common thread is that the merge list is the load-bearing data; the storage format just determines how it is serialized.

TL;DR

A merge rule is one entry in BPE's ordered learned list: 't' + 'h' = 'th'. Encoders replay merges greedily in learned order. Storage: merges.txt, tokenizer.json, or the tiktoken binary.

Memory aid

Easy to grasp

Imagine a cookbook where every recipe just says 'combine X with Y to make Z'. The cookbook has hundreds of recipes in a specific order. You start with a kitchen full of plain ingredients. To cook anything, you flip through the recipes in order and every time you see an ingredient pair you have in the kitchen, you combine them as the recipe says. Then you keep flipping. By the end, your kitchen has a few big assembled dishes instead of many small ingredients. BPE merges work the same way. The training process produced an ordered cookbook of pair-combine rules. The encoder reads the cookbook in order and combines every applicable pair in the input. Different tokenizer libraries print the cookbook in different file formats, but the cookbook itself is the same kind of object.

Key concepts

Byte Pair Encoding Merge Rule Tokenizer Artifact Encoding

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

A merge rule is the load-bearing unit of BPE tokenization. BPE training produces a vocabulary and an ordered list of merge rules; BPE encoding replays that list deterministically on every new input. Understanding what a merge rule is, where it is stored, and why the order matters is one of the cleanest ways to demonstrate that you understand what BPE actually does at inference time.

This deep dive walks through the training-time origin of merges, the encoding-time replay, the storage formats across the major 2026 tokenizer libraries, and the senior nuances that separate a careful answer from a superficial one.

Where merges come from: BPE training

BPE training is a simple loop that produces an ordered list of merge rules. The setup is:

Choose a base alphabet. For classical BPE this is the set of Unicode characters in the training corpus. For byte-level BPE (the GPT-2 variant used by tiktoken, Llama 3, and most modern generative LLMs) it is the 256 possible byte values.
Tokenize the training corpus into the base alphabet.
Repeat until the vocabulary reaches the target size: count every adjacent pair of pieces across the corpus, pick the pair with the highest frequency, merge every occurrence into a new piece, add the new piece to the vocabulary, and record the merge as a rule.

The output of training is two things. The vocabulary is the base alphabet plus every merged piece, mapped to integer ids. The merge list is an ordered sequence of pair-token rules: '(A, B) becomes AB'. The list is in the order the merges were learned, which is also (roughly) from most frequent to least frequent on the training corpus.

A typical merge list contains tens of thousands of entries. For a 100K-vocabulary byte-level BPE tokenizer like cl100k_base, the list has roughly 99,744 entries (100K total vocab minus 256 base bytes). For a 200K vocabulary like o200k_base it has roughly 199,744 entries.

The training process is corpus-dependent. The same algorithm on different corpora produces different vocabularies and different merge lists. The same algorithm on the same corpus with different vocabulary-size budgets produces lists of different lengths but the early merges are identical because they are picked by frequency before any cutoff.

How merges drive encoding

Where merges live: storage formats by library

What merges are not

Why this matters for interviews and for practice

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

GPT-2's original tokenizer ships with merges.txt and vocab.json files in the model directory; you can open merges.txt and read the BPE training history line by line.
Llama 3's tokenizer ships as tokenizer.json in the model repository on Hugging Face; the merges are nested inside the model.merges field of the JSON.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the order of the merge list critical and what would happen if you shuffled it?

Earlier merges create pieces that later merges combine. If 't' + 'h' becomes 'th' is at position 5 and 'th' + 'e' becomes 'the' is at position 12, shuffling so that 'the' merge comes first means it will never fire because 'th' does not yet exist. Random shuffling produces a tokenizer that mostly fails to merge anything beyond the simplest pieces, drastically inflating sequence length.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Thinking BPE encoding picks the highest-scoring merge at each step at inference time. Encoding is not re-running BPE training; it is replaying the learned merges in their training order, greedily, until no more apply.

60 second bullets to scan on the way to the call.

Define a merge rule as one entry in BPE's ordered pair-token list.
Explain why the order of the list matters for encoding.

Primary sources. Browse if you want the original framing.