Why does a 4-space Python indentation block often tokenize to a single token in cl100k_base?

MCQ·Medium·4.0 · 0·~1 min·Relevant atOpenAI

Attempt it

Quick reference

TL;DR

Four spaces are one of the most frequent byte sequences in Python source, so BPE merged them into a single dedicated token rather than four space tokens.

Memory aid

Easy to grasp

Imagine a stenographer who invents shorthand for whatever phrases come up most. After transcribing thousands of Python files, they notice 'four spaces at the start of a line' shows up constantly, so they create one quick squiggle for it instead of writing four marks every time. The tokenizer does the same thing: because Python uses four-space indentation everywhere, the four-space run became so common during training that it earned its own single symbol, called a token. That one squiggle now stands in for what used to be four.

Key concepts

Byte Pair Encoding Code tokenization BPE merge rules

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

2 min: BPE merges most-frequent pair + four-space run is ubiquitous in code training data + regex keeps the run intact + emergent not hardcoded + tabs/odd widths fragment.

Code example

python

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

# A single level of Python indentation: four spaces
print(enc.encode("    "))      # often one token id

# Inside real code, the indent stays cheap
src = "def f(n):\n    return n + 1\n"
print(len(enc.encode(src)))     # leading indent contributes ~1 token

# Non-standard indentation fragments
print(enc.encode("   "))        # three spaces: may be 1-3 tokens
print(enc.encode("\t"))         # a tab: separate entry, different cost

Real-world examples

Real products, models, and research that use this idea.

tiktoken with cl100k_base encodes a four-space Python indent as a single token, visible in the OpenAI tokenizer playground.
Code assistants like Cursor and GitHub Copilot rely on this whitespace efficiency to fit more code in the context window.
Models such as GPT-5.5 and Claude Opus 4.7 inherit code-friendly tokenizers because their training mixes include large amounts of source code.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would the four-space behavior change if the tokenizer were trained mostly on prose instead of code?

QWhy might mixing tabs and spaces inflate token counts for the same logical program?

QHow would you confirm empirically that a given indent width is a single token?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming the tokenizer has a hardcoded rule for indentation. It does not; the four-space token emerges purely from how often that byte sequence appeared in the training data.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.