Why does a 4 space Python indentation block often tokenize to a single token in cl100k_base?
Four spaces are one of the most frequent byte sequences in Python source, so BPE merged them into a single dedicated token rather than four space tokens.
Imagine a stenographer who invents shorthand for whatever phrases come up most. After transcribing thousands of Python files, they notice 'four spaces at the start of a line' shows up constantly. So they create one quick squiggle for it instead of writing four marks every time. The tokenizer does the same thing. Python uses four-space indentation everywhere, so the four-space run became so common during training that it earned its own single symbol, called a token. That one squiggle now stands in for what used to be four.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Paste a Python function into a tokenizer viewer, and the four-space indent at the start of a line is a single token. That is the same cost as a short word. There is no syntax engine inside the tokenizer that understands Python indentation. So where does the single token come from?
Subword tokenizers are statistical compressors trained on a corpus, and whatever appears often gets compressed hard. Python's four-space indentation is one of the most repeated byte patterns in any training set with a lot of code. So it gets the same treatment a common English word does.
Each wrong option is a tempting half-truth. There is no special indent mode. The regex contributes but does not act alone. And spaces are not free. Each is a real misconception that someone has shipped a cost estimate on.
How BPE turns frequency into vocabulary
Byte Pair Encoding starts with the raw byte alphabet, just 256 symbols. It scans the training corpus, counts every adjacent pair of current symbols, and merges the single most frequent pair into a new symbol. Then it repeats, thousands of times, until the vocabulary reaches its target size.
The rule is purely greedy and frequency-based. There is no grammar, no notion of words or indentation. A pair wins a merge only if it is currently the most common adjacent pair in the data.
This is why the training mix matters so much. A pattern that is rare gets no merges and stays fragmented into bytes. A pattern that is everywhere climbs to the top of the frequency table again and again. It picks up merge after merge until it becomes a single dense token.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
# A single level of Python indentation: four spaces
print(enc.encode(" ")) # often one token id
# Inside real code, the indent stays cheap
src = "def f(n):\n return n + 1\n"
print(len(enc.encode(src))) # leading indent contributes ~1 token
# Other indent styles also have entries in cl100k_base
print(enc.encode(" ")) # three spaces: one token id
print(enc.encode("\t")) # a tab: one (different) token idReal products, models, and research that use this idea.
- tiktoken with cl100k_base encodes a four-space Python indent as a single token, visible in the OpenAI tokenizer playground.
- Code assistants like Cursor and GitHub Copilot rely on this whitespace efficiency to fit more code in the context window.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would the four-space behavior change if the tokenizer were trained mostly on prose instead of code?
Tie it back to merge frequency; without a code-heavy corpus the four-space run loses its high-frequency rank and may not earn a dedicated merge.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming the tokenizer has a hardcoded rule for indentation. It does not; the four-space token emerges purely from how often that byte sequence appeared in the training data.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.