Why does a 4-space Python indentation block often tokenize to a single token in cl100k_base?

Attempt it

Quick reference

TL;DR

Four spaces are one of the most frequent byte sequences in Python source, so BPE merged them into a single dedicated token rather than four space tokens.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a stenographer who invents shorthand for whatever phrases come up most. After transcribing thousands of Python files, they notice 'four spaces at the start of a line' shows up constantly, so they create one quick squiggle for it instead of writing four marks every time. The tokenizer does the same thing: because Python uses four-space indentation everywhere, the four-space run became so common during training that it earned its own single symbol, called a token. That one squiggle now stands in for what used to be four.

Key concepts

Byte Pair Encoding Code tokenization BPE merge rules

Quick scan · 2–4 min

Everything important, quickly.

2 min: BPE merges most-frequent pair + four-space run is ubiquitous in code training data + regex keeps the run intact + emergent not hardcoded + tabs/odd widths fragment.

Code example

python

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

# A single level of Python indentation: four spaces
print(enc.encode("    "))      # often one token id

# Inside real code, the indent stays cheap
src = "def f(n):\n    return n + 1\n"
print(len(enc.encode(src)))     # leading indent contributes ~1 token

# Non-standard indentation fragments
print(enc.encode("   "))        # three spaces: may be 1-3 tokens
print(enc.encode("\t"))         # a tab: separate entry, different cost

Real-world examples

Real products, models, and research that use this idea.

tiktoken with cl100k_base encodes a four-space Python indent as a single token, visible in the OpenAI tokenizer playground.
Code assistants like Cursor and GitHub Copilot rely on this whitespace efficiency to fit more code in the context window.
Models such as GPT-5.5 and Claude Opus 4.7 inherit code-friendly tokenizers because their training mixes include large amounts of source code.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would the four-space behavior change if the tokenizer were trained mostly on prose instead of code?

Sign in or go Pro to see the approach for this follow-up.

QWhy might mixing tabs and spaces inflate token counts for the same logical program?

Sign in or go Pro to see the approach for this follow-up.

QHow would you confirm empirically that a given indent width is a single token?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.