What are BOS, EOS, PAD, and role marker special tokens, and what happens when each is missing or duplicated? · Zenaique

Attempt it

BOS (beginning of sequence): initializes the autoregressive state; missing BOS can degrade generation quality since the model's initial hidden state was trained to start from the BOS embedding.

EOS (end of sequence): signals the model to stop generating; missing EOS causes infinite generation loops.

PAD (padding): used to align batches to equal length; must be masked in attention or it corrupts attention scores.

Role markers (e.g., [INST], <|system|>, <|user|>): define conversation structure in instruction-tuned models; wrong or missing markers cause role confusion or prompt leakage.

Your answer

Quick reference

TL;DR

BOS starts generation, EOS stops it (infinite loop if missing), PAD aligns batches and must be attention-masked, and role markers fence off who speaks in a chat.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a stage play where the script has stage directions that nobody reads aloud. BOS is the 'curtain up, scene starts here' note. EOS is the 'curtain down, stop here' note, and without it the actor keeps improvising forever. PAD is blank filler added so every actor's script has the same number of pages, and the director has to remember not to read the blank pages out loud. Role markers are the name tags pinned on each speaker so the audience knows when the host, the guest, or the announcer is talking. Lose a name tag and the audience can no longer tell who said what.

Key concepts

Special tokens Chat templates

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

2 min: BOS start state + EOS as learned stop + PAD attention masking + role markers as trust boundaries + apply_chat_template as the safe path.

Real-world examples

Real products, models, and research that use this idea.

Llama 4 Maverick served locally needs the tokenizer's add_bos_token config set correctly, since base-model quality drops when BOS is missing from the prompt.
OpenAI fine-tunes for GPT-5.5 use the ChatML format where <|im_start|>system and <|im_end|> fence each role, and malformed markers break the role contract.
vLLM's continuous batching depends on correct PAD masking, since a wrong mask can let one request attend across into another request's padded slots.
Anthropic's Claude Opus 4.7 API hides BOS and EOS from callers entirely, so users send structured messages instead of raw strings, sidestepping marker bugs.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the attention mask mechanically stop PAD tokens from earning attention weight?

Sign in or go Pro to see the approach for this follow-up.

QWhat is Special Token Injection, and which tokenizer flag is the main defense?

Sign in or go Pro to see the approach for this follow-up.

QHow does a model learn to emit EOS at the right moment during training?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Saying PAD tokens are ignored automatically. They are not. You must pass an explicit attention mask, or the softmax spends weight on empty positions.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.

Hugging Face: Chat templates and special tokens