BOS, EOS, PAD: why do tokenizers reserve these ids instead of using normal text?

Flashcard·Easy·4.0 · 0·~30s·Asked atAi21 Tata Digital Zepto·Relevant atMeta

Attempt it

Because the model needs unambiguous boundary signals that can never be confused with actual content. If you used the word 'END' as your stop signal, any text containing 'END' would falsely trigger it. Reserved ids like BOS, EOS, and PAD solve this: they occupy their own vocabulary positions, are never produced by tokenizing normal user text, and the model learned during training that these specific ids mark structural boundaries. The same logic extends to chat role markers like <|im_start|> and [INST].

Your answer

Tokenizers reserve these ids because the model needs unambiguous boundary signals that can never collide with real content. If you tried to use the word 'END' as a stop signal, any input containing 'END' would falsely trigger it. **Special tokens** solve this by occupying dedicated vocabulary positions that normal tokenization never produces. The standard set covers four categories. **BOS** (beginning of sequence) tells the model where input starts. **EOS** (end of sequence) is the signal the model emits when done; the runtime watches for this id and stops decoding. **PAD** fills shorter sequences in a batch to a common length so attention masks can ignore them. **Role markers** like `<|im_start|>`, `<|im_end|>`, `[INST]`, and `[/INST]` mark conversational turns in chat templates. These ids work because the model learned to treat them as structural boundaries during training. Billions of training examples had EOS at the end, so the model internalized that emitting EOS means completion. Ordinary content tokens never had this training signal. Tokenizing normal text like 'Hello world' produces no special tokens. They are inserted only by the chat template or when add_special_tokens=True is set. If a user types the literal string `<|im_start|>` and the tokenizer maps it to the role marker id, that is a prompt injection vulnerability, which is why production systems sanitize input.

**Special tokens** are reserved vocabulary entries carrying structural rather than content meaning. They occupy ids in the same numbered vocabulary as ordinary tokens, but their purpose is to mark boundaries, signal control flow, or fill space in batches. The standard categories vary by model family but the roles are stable. **BOS** marks sequence start; some models always prepend one, some never do. **EOS** signals the model to stop; sampling loops check for this id and exit. **PAD** fills shorter batch members to a common length; PAD positions are masked from attention and loss. **Role markers** like `<|im_start|>`/`<|im_end|>` (OpenAI template), `[INST]`/`[/INST]` (Mistral/Llama 2), `<|start_header_id|>` (Llama 3) mark chat turns. **Tool-use markers** in tool-calling models structure function requests and results. The load-bearing property: the model learned these ids during training. The lm_head and embedding rows for EOS were trained on sequences where EOS marked actual stopping points. The model internalized that emitting EOS means done. Using a random vocabulary id as a stop signal would not work because the model has no learned association with it. Special tokens normally do not appear in user-text tokenization. The tokenizer's encode method has an add_special_tokens parameter, and chat-template libraries insert role markers when applying the template. The boundary between structural insertion and raw tokenization is where prompt injection bugs live: if user input flows through with role marker strings unsanitized, the user has a backdoor into the chat structure.

TL;DR

Reserved ids prevent ambiguity: BOS, EOS, and PAD occupy positions normal tokenization never produces, so they can never collide with real content.

Memory aid

Easy to grasp

Imagine the tokenizer's vocabulary is mostly little pieces of words, but a few entries are not words at all. They are special markers, like invisible punctuation only the model can see: 'this is where the conversation starts', 'this is where this person stops talking', 'ignore this position, it is just padding'. These markers have their own numbered ids, but they never appear in ordinary text. The tokenizer inserts them when it knows it should, and the model learned during training that when it sees these ids, it should change behavior: stop generating, switch speaker, or skip the position. If a user accidentally types one of these marker strings as text, things can go wrong, which is why production systems sanitize input.

Key concepts

Special Token BOS EOS PAD Role Marker

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Most of a tokenizer's vocabulary is content: sub-word fragments the model uses to represent ordinary text. A small fraction is special: reserved ids carrying structural meaning. These special tokens are how the model knows when a conversation starts and ends, which speaker is talking, where a tool call begins, and when to switch output modes.

This deep dive defines special tokens precisely, walks the standard categories, explains how the model learned to interpret them, and covers the prompt injection risk that comes with letting user input touch them.

Special tokens defined precisely

Special tokens are reserved vocabulary entries carrying structural meaning rather than text content. They occupy ids in the same numbered vocabulary as content tokens. The embedding matrix has a row for each; the lm_head has a column. From a tensor perspective they are indistinguishable from any other token.

The distinction is semantic. Content tokens represent text fragments: 'token', 'ization', ' the'. When the model emits one, it is generating part of an answer. Special tokens represent boundaries or control signals: 'this sequence is starting', 'I am done generating', 'a tool call begins'. When the model emits one, the runtime reacts: stopping the sampling loop, parsing a tool call, switching roles.

The ids are arbitrary. cl100k_base assigns <|endoftext|> to id 100257 by convention. Any id would have worked with the same training. What matters is consistency between training and inference.

Special tokens are typically defined when the tokenizer is created. Adding new ones later requires extending the embedding matrix and lm_head, and the model has no learned behavior for them until fine-tuning teaches it.

The standard categories

How the model learned to interpret them

The prompt injection risk

Practical guidance for working with special tokens

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

OpenAI's cl100k_base reserves ids like 100257 for `<|endoftext|>` and 100264 for `<|im_start|>`; the chat API applies the template to insert these around each message.
Llama 3 and Llama 4 use `<|begin_of_text|>`, `<|start_header_id|>`, `<|end_header_id|>`, and `<|eot_id|>` as chat template tokens; exact formatting matters for quality.

What an interviewer would ask next. Try answering before peeking at the approach.

QThe model just emits ids. What actually stops generation when it produces EOS?

The sampling loop in the runtime checks each emitted id against a configured stop token set. On match, the loop exits. The model would happily emit more tokens after EOS; the runtime breaks. This is why misconfigured stop sets cause runaway generation.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Letting raw user input pass through the tokenizer with add_special_tokens=True, allowing a user to inject role markers like `<|im_start|>` and steer the model past the system prompt.

60 second bullets to scan on the way to the call.

Define a special token as a reserved vocabulary entry carrying structural rather than content meaning.
Name the standard categories: sentinels (BOS, EOS), PAD, role markers, tool-use markers.

Primary sources. Browse if you want the original framing.