BPE learned 50,000 merge rules during training. What is a merge rule, and where does it live at inference?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

A merge rule is one entry in the ordered list BPE learned during training: 't' + 'h' becomes 'th', then 'th' + 'e' becomes 'the', and so on. At encoding time the tokenizer applies the merges greedily in their learned order to produce the final token sequence. Merges live in the tokenizer artifact: a merges.txt file for GPT-2-style tokenizers, embedded inside tokenizer.json for HuggingFace fast tokenizers, and embedded in the binary encoder for tiktoken.

Your answer

**A merge rule** is one entry in the ordered list that BPE produced during training. Each entry says 'two specific pieces combine into one new piece'. The list might start with 't' + 'h' becomes 'th', then continue with 'in' + 'g' becomes 'ing', then 'th' + 'e' becomes 'the', and so on, for tens of thousands of entries. At encoding time, the tokenizer applies these merges to any input string. It splits the input into the base alphabet (bytes or characters), then walks through its merge list in learned order and applies every applicable merge greedily. Once no more merges apply, the resulting sequence of pieces is the tokenization. The order is critical because earlier merges create the pieces that later merges combine. Where the list lives depends on the library: - **GPT-2-style tokenizers** store merges in a plain text file called merges.txt, one merge per line. - **HuggingFace fast tokenizers** embed the merges (along with vocabulary and pre-tokenization rules) inside a single tokenizer.json file. - **tiktoken** packs the merges into a compact binary encoder that ships with the OpenAI Python package; you do not see them as a separate file. - **SentencePiece BPE** packs them into the .model protobuf file alongside the vocabulary. In 2026 the common format is tokenizer.json for non-OpenAI models and the tiktoken binary for OpenAI models.

**A merge rule** is one entry in the ordered list of pair-token combinations that BPE produced during training. The list is the core artifact of BPE; everything else (the vocabulary itself) can be derived from it given a base alphabet. During training, BPE starts from a base alphabet (bytes for byte-level BPE, characters for classical BPE) and iteratively picks the most frequent adjacent pair across the corpus, merges every occurrence into a new piece, and records the merge as a rule. The output of training is a vocabulary (the base alphabet plus every merged piece) and an ordered merge list. At encoding time, the tokenizer applies the merges greedily in the order they were learned. The pseudocode is: 1. Split the input string into the base alphabet (bytes or characters). 2. Walk the merge list in order. For each rule '(A, B) becomes AB', scan the current piece sequence and merge every adjacent occurrence of A followed by B into AB. 3. Continue until the merge list is exhausted. The order matters because earlier merges create the pieces that later merges combine. Encoding is not a search for the optimal segmentation; it is a deterministic replay of the learned merges. The storage format varies by library: - **GPT-2-style and early HuggingFace tokenizers**: a plain-text merges.txt file with one merge per line (space-separated pair) plus a separate vocab.json mapping each token to its id. - **HuggingFace fast tokenizers (2026 standard)**: a single tokenizer.json file that encodes the pre-tokenizer, the model (including merges and vocab), the post-processor, and the decoder. - **tiktoken**: the merges and vocabulary are packed into a compact binary blob that the encoder reads at load time. You do not see merges.txt; the list is in the binary. - **SentencePiece BPE**: merges and vocabulary live inside the .model protobuf file. The common thread is that the merge list is the load-bearing data; the storage format just determines how it is serialized.

Quick reference

TL;DR

A merge rule is one entry in BPE's ordered learned list: 't' + 'h' = 'th'. Encoders replay merges greedily in learned order. Storage: merges.txt, tokenizer.json, or the tiktoken binary.

Memory aid

Easy to grasp

Imagine a cookbook where every recipe just says 'combine X with Y to make Z'. The cookbook has hundreds of recipes in a specific order. You start with a kitchen full of plain ingredients. To cook anything, you flip through the recipes in order and every time you see an ingredient pair you have in the kitchen, you combine them as the recipe says. Then you keep flipping. By the end, your kitchen has a few big assembled dishes instead of many small ingredients. BPE merges work the same way. The training process produced an ordered cookbook of pair-combine rules. The encoder reads the cookbook in order and combines every applicable pair in the input. Different tokenizer libraries print the cookbook in different file formats, but the cookbook itself is the same kind of object.

Key concepts

Byte Pair Encoding Merge Rule Tokenizer Artifact Encoding

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: define a merge rule as one entry in BPE's ordered list, walk through how encoding replays the list greedily, explain why the order matters, and list the storage formats in three or four major tokenizer libraries.

Real-world examples

Real products, models, and research that use this idea.

GPT-2's original tokenizer ships with merges.txt and vocab.json files in the model directory; you can open merges.txt and read the BPE training history line by line.
Llama 3's tokenizer ships as tokenizer.json in the model repository on Hugging Face; the merges are nested inside the model.merges field of the JSON.
tiktoken's cl100k_base and o200k_base are binary encoder files included in the tiktoken package; you cannot inspect the merges directly without reverse-engineering the binary.
SentencePiece-BPE models for Mistral ship as tokenizer.model files (the SentencePiece protobuf); the merges and vocabulary live in the same file.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the order of the merge list critical and what would happen if you shuffled it?

QHow does BPE encoding scale with merge-list length?

QCan you reconstruct the vocabulary from the merge list alone?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Thinking BPE encoding picks the highest-scoring merge at each step at inference time. Encoding is not re-running BPE training; it is replaying the learned merges in their training order, greedily, until no more apply.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.