Predict whether tokenizer.encode('world') and tokenizer.encode(' world') produce the same token IDs in tiktoken.

Attempt it

Using tiktoken with cl100k_base:

```python
import tiktoken
enc = tiktoken.get_encoding('cl100k_base')
print(enc.encode('world'))
print(enc.encode(' world'))
```

Predict whether the two calls produce the same list of token IDs. If different, explain why.

Quick reference

TL;DR

Different IDs: byte-level BPE folds the leading space into the token, so ' world' and 'world' are distinct entries with distinct IDs.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine your phone's autocomplete keeps two separate buttons: one for the word right after a space, like the ' apple' you tap mid-sentence, and one for 'apple' stuck to the start of a line. They look almost the same to you, but the keyboard files them in different slots. A byte-level tokenizer does the same thing. The space in front of a word is glued onto the word and becomes part of which button gets pressed, so the version with a space and the version without get different ID numbers.

Key concepts

Byte Pair Encoding tiktoken

Quick scan · 2–4 min

Everything important, quickly.

2 min: different IDs + space folded into the token + frequency reason + lossless decode + the concatenation bug + the apply_chat_template guard.

Code example

python

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")

print(enc.encode("world"))     # e.g. [14957]  -> no leading space
print(enc.encode(" world"))    # e.g. [1917]    -> leading space folded in

# Same letters, different IDs. Decode round-trips the space:
print(repr(enc.decode([1917])))  # ' world'

# The trap: gluing pieces without spaces shifts the boundary token.
print(enc.encode("hello" + "world"))   # 'world' has no leading space
print(enc.encode("hello" + " world"))  # natural spacing

Real-world examples

Real products, models, and research that use this idea.

OpenAI's tiktoken cl100k_base and o200k_base both fold the leading space into tokens, which is why the GPT-5.5 tokenizer playground shows ' world' and 'world' as separate cells.
GPT-2's original byte-level BPE introduced the leading-space convention that tiktoken and most modern tokenizers inherited.
Llama and Mistral SentencePiece tokenizers represent the leading space as a visible underscore meta-symbol, the same boundary idea by a different surface form.
HuggingFace's add_prefix_space option exists precisely so token-classification users can control whether the first word gets its space-prefixed variant.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does SentencePiece represent the same leading-space distinction, and how is it different from byte-level BPE?

Sign in or go Pro to see the approach for this follow-up.

QIf ' world' and 'world' differ, how does the model ever learn they mean the same word?

Sign in or go Pro to see the approach for this follow-up.

QWhat is the safest way to build multi-part chat prompts given this quirk?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.