Why does a 4 space Python indentation block often tokenize to a single token in cl100k_base?
Four spaces are one of the most frequent byte sequences in Python source, so BPE merged them into a single dedicated token rather than four space tokens.
Imagine a stenographer who invents shorthand for whatever phrases come up most. After transcribing thousands of Python files, they notice 'four spaces at the start of a line' shows up constantly, so they create one quick squiggle for it instead of writing four marks every time. The tokenizer does the same thing: because Python uses four-space indentation everywhere, the four-space run became so common during training that it earned its own single symbol, called a token. That one squiggle now stands in for what used to be four.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
It surprises people the first time they see it: paste a Python function into a tokenizer viewer and the four-space indent at the start of a line is a single token, the same cost as a short word. There is no syntax engine inside the tokenizer that understands Python indentation. So where does the single token come from?
The answer is a small but important lesson about how subword tokenizers actually work. They are statistical compressors trained on a corpus, and whatever appears often gets compressed hard. Python's four-space indentation is one of the most repeated byte patterns in any code-heavy training set, so it gets the same treatment a common English word does.
We will trace the BPE merge process that produces the token, see why the pre-tokenization regex lets it happen, and then look at where this efficiency quietly disappears so you do not over-trust it when budgeting code-generation tokens.
The distractors in the question are worth keeping in mind as we go, because each is a tempting half-truth. There is no special indent mode; the regex contributes but does not act alone; and spaces are emphatically not free. Each wrong answer is a real misconception that someone has shipped a cost estimate on, so the goal is not just to pick option A but to be able to say precisely why the other three fail.
How BPE turns frequency into vocabulary
Byte Pair Encoding starts with the raw byte alphabet, just 256 symbols. It scans the training corpus, counts every adjacent pair of current symbols, and merges the single most frequent pair into a new symbol. Then it repeats, thousands of times, until the vocabulary reaches its target size.
The rule is purely greedy and frequency-based. There is no grammar, no notion of words or indentation. A pair wins a merge only if it is currently the most common adjacent pair in the data.
This is why the training mix matters so much. A pattern that is rare gets no merges and stays fragmented into bytes. A pattern that is everywhere climbs to the top of the frequency table again and again, picking up merge after merge until it becomes a single dense token.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
# A single level of Python indentation: four spaces
print(enc.encode(" ")) # often one token id
# Inside real code, the indent stays cheap
src = "def f(n):\n return n + 1\n"
print(len(enc.encode(src))) # leading indent contributes ~1 token
# Non-standard indentation fragments
print(enc.encode(" ")) # three spaces: may be 1-3 tokens
print(enc.encode("\t")) # a tab: separate entry, different costReal products, models, and research that use this idea.
- tiktoken with cl100k_base encodes a four-space Python indent as a single token, visible in the OpenAI tokenizer playground.
- Code assistants like Cursor and GitHub Copilot rely on this whitespace efficiency to fit more code in the context window.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would the four-space behavior change if the tokenizer were trained mostly on prose instead of code?
Tie it back to merge frequency; without a code-heavy corpus the four-space run loses its high-frequency rank and may not earn a dedicated merge.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming the tokenizer has a hardcoded rule for indentation. It does not; the four-space token emerges purely from how often that byte sequence appeared in the training data.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.