A chat template wraps every turn with BOS and EOS tokens. Why does the model need them?
BOS anchors the input at the position-zero distribution the model was trained on; EOS is the termination signal the runtime checks to stop decoding. Omitting either breaks quality or causes runaway generation.
Think of a stage actor and a stage manager. The actor needs a curtain raising to know the scene begins, otherwise they start mid-sentence and the audience misses the setup. That curtain is the BOS token. When the scene ends, the actor bows, and that bow tells the stage manager to bring the lights down. That bow is the EOS token. The chat template is the stagehand who raises and lowers the curtain on cue, so the actor never handles the ropes. If you raise the curtain twice or forget it entirely, the performance looks wrong in a way that is hard to pin down but easy to notice.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
BOS and EOS look like trivia until something breaks in production. They are two specific token ids the model was trained with, and getting them right is a precondition for the model to behave as its model card promises.
This explanation walks through what each token is for, what the literal ids look like across the families you will actually deploy, how the chat template machinery hides the bookkeeping, and the two production bugs that come up most often.
What BOS is and why position zero matters
BOS is the beginning-of-sequence token, prepended once to the front of the model's input. The model was trained with this id at index zero of every example, which means the learned probability distribution at position zero is overwhelmingly the BOS token itself. Real content begins at position one.
The practical implication: omitting BOS shifts the first real content token to position zero, a slot the model was never trained on as content. The model still produces output, but it operates slightly out of distribution. Internal evals at large labs consistently show a small but measurable quality drop from omitted BOS, which is why every model card insists on it.
The literal token differs by family. Llama 3 and later use <|begin_of_text|>. Older SentencePiece families (Llama 2, Mistral 7B, Code Llama) use <s>. GPT-2 style models often do not use BOS at all. Always check the tokenizer config rather than guessing.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- HuggingFace transformers AutoTokenizer applies the model's chat template (Llama 3, Mistral, Qwen) and inserts BOS plus role markers automatically.
- vLLM and SGLang both read tokenizer.eos_token_id plus additional stop ids from the generation config when serving Llama 3 instruct.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat concretely goes wrong if you prepend BOS twice on a Llama 3 input?
The model sees the second BOS at position one, a position whose training distribution was real content. Outputs still look fluent but drift in tone, length, and instruction-following, and the drift is easy to miss without a regression eval. Decode the first ten ids of a real request to verify.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Calling apply_chat_template and then manually prepending BOS again. The template already inserts it, so the model sees two BOS tokens and operates out of distribution.
60 second bullets to scan on the way to the call.
State in one sentence what BOS marks and what EOS signals.
Name the literal Llama 3 BOS and EOS tokens versus the older SentencePiece equivalents.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.