BOS, EOS, PAD: why do tokenizers reserve these ids instead of using normal text?

Flashcard·Easy·4.0 · 0·~30s·Asked atAi21 Tata Digital Zepto·Relevant atMeta

Attempt it

Because the model needs unambiguous boundary signals that can never be confused with actual content. If you used the word 'END' as your stop signal, any text containing 'END' would falsely trigger it. Reserved ids like BOS, EOS, and PAD solve this. They occupy their own vocabulary positions, and tokenizing normal user text never produces them. The model learned during training that these specific ids mark structural boundaries. The same logic extends to chat role markers like <|im_start|> and [INST].

Your answer

Tokenizers reserve these ids because the model needs clear boundary signals that can never clash with real content. If you used the word 'END' as a stop signal, any input containing 'END' would falsely trigger it. **Special tokens** solve this by using their own vocabulary positions that normal tokenization never produces. The standard set covers four categories. **BOS** (beginning of sequence) tells the model where input starts. **EOS** (end of sequence) is the signal the model emits when done; the runtime watches for this id and stops decoding. **PAD** fills shorter sequences in a batch to a common length so attention masks can ignore them. **Role markers** like `<|im_start|>` and `<|im_end|>` mark conversational turns in chat templates. These ids work because the model learned to treat them as structural boundaries during training. Billions of training examples ended with EOS, so the model learned that emitting EOS means completion. Tokenizing normal text like 'Hello world' produces no special tokens. Only the chat template or add_special_tokens=True (for BOS and EOS) inserts them. If a user types the literal string `<|im_start|>` and the tokenizer maps it to the role marker id, that is a prompt injection vulnerability. So production systems sanitize input.

**Special tokens** are reserved vocabulary entries carrying structural rather than content meaning. They occupy ids in the same numbered vocabulary as ordinary tokens, but their purpose is to mark boundaries, signal control flow, or fill space in batches. The standard categories vary by model family but the roles are stable. **BOS** marks sequence start; some models always prepend one, some never do. **EOS** signals the model to stop; sampling loops check for this id and exit. **PAD** fills shorter batch members to a common length; PAD positions are masked from attention and loss. **Role markers** like `<|im_start|>`/`<|im_end|>` (OpenAI template), `[INST]`/`[/INST]` (newer Mistral), `<|start_header_id|>` (Llama 3) mark chat turns. **Tool use markers** in tool calling models structure function requests and results. The model learned these ids during training. The lm_head and embedding rows for EOS were trained on sequences where EOS marked actual stopping points. The model learned that emitting EOS means done. Using a random vocabulary id as a stop signal would not work because the model has no learned association with it. Special tokens normally do not appear in the tokenization of user text. The tokenizer's add_special_tokens parameter controls BOS and EOS, and chat template libraries insert role markers when applying the template. Prompt injection bugs live at the boundary between structural insertion and raw tokenization. If user input flows through with role marker strings unsanitized, the user has a backdoor into the chat structure.

TL;DR

Reserved ids prevent ambiguity: BOS, EOS, and PAD occupy positions normal tokenization never produces, so they can never collide with real content.

Memory aid

Easy to grasp

Imagine the tokenizer's vocabulary is mostly little pieces of words, but a few entries are not words at all. They are special markers, like invisible punctuation only the model can see: 'this is where the conversation starts', 'this is where this person stops talking', 'ignore this position, it is just padding'. These markers have their own numbered ids, but they never appear in ordinary text. The tokenizer inserts them when it knows it should. During training, the model learned to change behavior when it sees these ids: stop generating, switch speaker, or skip the position. If a user accidentally types one of these marker strings as text, things can go wrong. That is why production systems sanitize input.

Key concepts

Special Token BOS EOS PAD Role Marker

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Most of a tokenizer's vocabulary is content: sub-word fragments the model uses to represent ordinary text. A small fraction is special: reserved ids carrying structural meaning. These special tokens are how the model knows when a conversation starts and ends, which speaker is talking, where a tool call begins, and when to switch output modes.

Special tokens defined precisely

Special tokens are reserved vocabulary entries carrying structural meaning rather than text content. They occupy ids in the same numbered vocabulary as content tokens. The embedding matrix has a row for each; the lm_head has a column. As tensors, they look the same as any other token.

The distinction is semantic. Content tokens represent text fragments: 'token', 'ization', ' the'. When the model emits one, it is generating part of an answer. Special tokens represent boundaries or control signals: 'this sequence is starting', 'I am done generating', 'a tool call begins'. When the model emits one, the runtime reacts: stopping the sampling loop, parsing a tool call, switching roles.

The ids are arbitrary. cl100k_base assigns <|endoftext|> to id 100257 by convention. Any id would have worked with the same training. What matters is consistency between training and inference.

Special tokens are typically defined when the tokenizer is created. Adding new ones later requires extending the embedding matrix and lm_head. The model has no learned behavior for them until fine-tuning teaches it.

The standard categories

How the model learned to interpret them

The prompt injection risk

Practical guidance for working with special tokens

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

OpenAI's cl100k_base reserves id 100257 for <|endoftext|>, and a ChatML extension of it adds <|im_start|> at 100264; the chat API inserts its message markers itself.
Llama 3 uses <|begin_of_text|>, <|start_header_id|>, <|end_header_id|>, and <|eot_id|> as chat template tokens (Llama 4 renames them <|header_start|>, <|header_end|>, and <|eot|>); exact formatting matters for quality.

What an interviewer would ask next. Try answering before peeking at the approach.

QThe model just emits ids. What actually stops generation when it produces EOS?

The sampling loop in the runtime checks each emitted id against a configured stop token set. On match, the loop exits. The model would emit more tokens after EOS; the runtime breaks the loop. This is why misconfigured stop sets cause runaway generation.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Tokenizing raw user input without split_special_tokens=True, allowing a user to inject role markers like <|im_start|> and steer the model past the system prompt.

60 second bullets to scan on the way to the call.

Define a special token as a reserved vocabulary entry carrying structural rather than content meaning.
Name the standard categories: sentinels (BOS, EOS), PAD, role markers, tool-use markers.

Primary sources. Browse if you want the original framing.