BOS, EOS, PAD: why do tokenizers reserve these ids instead of using normal text? · Zenaique

Attempt it

Because the model needs unambiguous boundary signals that can never be confused with actual content. If you used the word 'END' as your stop signal, any text containing 'END' would falsely trigger it. Reserved ids like BOS, EOS, and PAD solve this: they occupy their own vocabulary positions, are never produced by tokenizing normal user text, and the model learned during training that these specific ids mark structural boundaries. The same logic extends to chat role markers like <|im_start|> and [INST].

Your answer

Tokenizers reserve these ids because the model needs unambiguous boundary signals that can never collide with real content. If you tried to use the word 'END' as a stop signal, any input containing 'END' would falsely trigger it. **Special tokens** solve this by occupying dedicated vocabulary positions that normal tokenization never produces. The standard set covers four categories. **BOS** (beginning of sequence) tells the model where input starts. **EOS** (end of sequence) is the signal the model emits when done; the runtime watches for this id and stops decoding. **PAD** fills shorter sequences in a batch to a common length so attention masks can ignore them. **Role markers** like `<|im_start|>`, `<|im_end|>`, `[INST]`, and `[/INST]` mark conversational turns in chat templates. These ids work because the model learned to treat them as structural boundaries during training. Billions of training examples had EOS at the end, so the model internalized that emitting EOS means completion. Ordinary content tokens never had this training signal. Tokenizing normal text like 'Hello world' produces no special tokens. They are inserted only by the chat template or when add_special_tokens=True is set. If a user types the literal string `<|im_start|>` and the tokenizer maps it to the role marker id, that is a prompt injection vulnerability, which is why production systems sanitize input.

**Special tokens** are reserved vocabulary entries carrying structural rather than content meaning. They occupy ids in the same numbered vocabulary as ordinary tokens, but their purpose is to mark boundaries, signal control flow, or fill space in batches. The standard categories vary by model family but the roles are stable. **BOS** marks sequence start; some models always prepend one, some never do. **EOS** signals the model to stop; sampling loops check for this id and exit. **PAD** fills shorter batch members to a common length; PAD positions are masked from attention and loss. **Role markers** like `<|im_start|>`/`<|im_end|>` (OpenAI template), `[INST]`/`[/INST]` (Mistral/Llama 2), `<|start_header_id|>` (Llama 3) mark chat turns. **Tool-use markers** in tool-calling models structure function requests and results. The load-bearing property: the model learned these ids during training. The lm_head and embedding rows for EOS were trained on sequences where EOS marked actual stopping points. The model internalized that emitting EOS means done. Using a random vocabulary id as a stop signal would not work because the model has no learned association with it. Special tokens normally do not appear in user-text tokenization. The tokenizer's encode method has an add_special_tokens parameter, and chat-template libraries insert role markers when applying the template. The boundary between structural insertion and raw tokenization is where prompt injection bugs live: if user input flows through with role marker strings unsanitized, the user has a backdoor into the chat structure.

Quick reference

TL;DR

Reserved ids prevent ambiguity: BOS, EOS, and PAD occupy positions normal tokenization never produces, so they can never collide with real content.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine the tokenizer's vocabulary is mostly little pieces of words, but a few entries are not words at all. They are special markers, like invisible punctuation only the model can see: 'this is where the conversation starts', 'this is where this person stops talking', 'ignore this position, it is just padding'. These markers have their own numbered ids, but they never appear in ordinary text. The tokenizer inserts them when it knows it should, and the model learned during training that when it sees these ids, it should change behavior: stop generating, switch speaker, or skip the position. If a user accidentally types one of these marker strings as text, things can go wrong, which is why production systems sanitize input.

Key concepts

Special Token BOS EOS PAD Role Marker

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define special tokens as reserved structural vocabulary entries, name the categories (sentinels, PAD, role markers, tool-use markers), explain why the model treats them as boundaries via training, cite chat-template tokens for two model families, and flag the prompt injection vector.

Real-world examples

Real products, models, and research that use this idea.

OpenAI's cl100k_base reserves ids like 100257 for `<|endoftext|>` and 100264 for `<|im_start|>`; the chat API applies the template to insert these around each message.
Llama 3 and Llama 4 use `<|begin_of_text|>`, `<|start_header_id|>`, `<|end_header_id|>`, and `<|eot_id|>` as chat template tokens; exact formatting matters for quality.
Mistral chat models use `[INST]` and `[/INST]` to wrap user instructions; Mistral tool-use models add additional tokens for function calls.
HuggingFace transformers exposes tokenizer.apply_chat_template() to insert the correct special tokens for any chat-tuned model.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QThe model just emits ids. What actually stops generation when it produces EOS?

Sign in or go Pro to see the approach for this follow-up.

QHow does prompt injection via special tokens work in practice?

Sign in or go Pro to see the approach for this follow-up.

QWhy do chat templates vary across model families, and what happens with the wrong one?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Letting raw user input pass through the tokenizer with add_special_tokens=True, allowing a user to inject role markers like `<|im_start|>` and steer the model past the system prompt.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.