A chat template wraps every turn with BOS and EOS tokens. Why does the model need them?

The model needs BOS and EOS for two different reasons, one at each end of the pipeline. **BOS** exists because the model was trained with a specific token at position zero of every training example. The learned distribution at position zero is overwhelmingly the BOS token, and real content starts at position one. Omitting BOS pushes the first content token into position zero, a slot the model was never trained on as content, and quality degrades subtly but consistently. For Llama 3 the literal token is `<|begin_of_text|>`; for older SentencePiece models (Llama 2, Mistral) it is `<s>`. **EOS** exists because the runtime needs a termination signal. After each generated token, the sampling loop checks whether the model just emitted the EOS id. If it did, decoding stops. Without EOS in the stop set, the model generates until the token limit, often hallucinating extra turns. For Llama 3, the stop set includes both `<|end_of_text|>` and the per-turn `<|eot_id|>`. In a chat template, both are handled automatically. `tokenizer.apply_chat_template(messages, add_generation_prompt=True)` inserts BOS once at the front, places role markers between turns, and positions the input so the assistant generates up to EOS. The most common bug is calling this and then manually prepending BOS, producing a double-BOS the model has never seen.

BOS and EOS are special tokens with two different jobs. **BOS** marks the start of the model's input; **EOS** is what the model emits to mark the end of its generation. BOS matters because the model was trained with it at position zero of every example. The first content token always appeared at position one. Omitting BOS shifts content into position zero, which is a distribution the model was never trained on, and quality degrades silently. Concrete ids: Llama 3 uses `<|begin_of_text|>`, older SentencePiece models (Llama 2, Mistral) use `<s>`. EOS is how generation stops. After each step the runtime compares the newest token id against the model's eos_token_id set and halts on match. Llama 3 uses `<|end_of_text|>` plus `<|eot_id|>` (per-turn end); the generation config lists both. Older models use `</s>`. Inside a chat template, both are inserted automatically. `tokenizer.apply_chat_template(messages, add_generation_prompt=True)` emits BOS once, role markers between turns, and positions the input for assistant generation. Do not add BOS or EOS manually on top; doubling them is a common bug that degrades generation in subtle ways.