A developer uses HuggingFace's tokenizer.apply_chat_template(messages, add_special_tokens=True) and then manually prepends tokenizer.bos_token_id to the resulting token IDs. Describe the failure mode this introduces, why it occurs, and how it affects model output quality.
Free · 2 AI evals / day
Most chat models expect their input to begin with one special **BOS token**, short for beginning-of-sequence. It marks 'the conversation starts here'.
When you call `apply_chat_template(messages, add_special_tokens=True)`, the tokenizer already places that single BOS at the front for you. If the developer then prepends another BOS by hand, the sequence now starts with two BOS tokens instead of one.
The problem is that the model was instruction-tuned only on sequences with exactly one BOS at the very start, immediately followed by the first role marker (like the system tag). It never saw two BOS tokens in a row during training.
So every token gets shifted one position to the right, and the spot where the model expects the system role marker now holds a second BOS instead. The model can misread where the system instructions begin and end. In practice this means the system prompt may be partly or fully ignored, and the model can confuse which turn it is reading. The fix is simple: do not add BOS yourself. Let the template handle it.
The failure is a **double-BOS sequence** the model never trained on, and it corrupts role parsing.
The call `apply_chat_template(..., add_special_tokens=True)` already inserts one BOS at position 0, which is exactly what the model expects. Prepending `tokenizer.bos_token_id` manually adds a second one, so the sequence opens with two BOS tokens.
Why that hurts: the model was instruction-tuned on sequences with one BOS at position 0 followed immediately by the first role marker, often the system tag. With a duplicate, every later token shifts right by one. The position where the model expects the system marker now holds a second BOS, and the structural signal it learned to read is broken.
The consequences are concrete. The system prompt may be misinterpreted, since attention at the system marker now points at an unexpected token. Role confusion appears, because the model cannot cleanly tell where the system turn ends and the user turn begins. In bad cases the system prompt is partially or fully ignored. One context position is also wasted, which is minor next to the structural damage.
The fix has two forms. Either set `add_special_tokens=False` when adding BOS yourself, or better, pass `add_generation_prompt=True` and let the template own all special tokens.
Quick reference
TL;DR
A second BOS shifts every token by one and is a sequence the model never trained on, corrupting role boundaries so the system prompt can be ignored or misparsed.
Imagine a song that always starts with one drumbeat, then the singer comes in. The band rehearsed it that way thousands of times: one beat, then sing. Now someone sneaks in a second drumbeat at the very start. The singer, trained to start right after the single beat, gets thrown off, comes in at the wrong moment, and the whole intro feels off.
The BOS token is that opening drumbeat for a chat model. The template already plays exactly one. Prepend another by hand and the model hears two beats it never practiced, so it misjudges where the first speaker, the system instructions, actually begins, and may drift or ignore them.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A pipeline where the tokenizer or template is configured to not add BOS Then a single manual prepend is correct, so the double-BOS warning does not apply and you must add it yourself.
Base-model raw completion outside any chat format There is no role structure to disrupt, so BOS handling follows the completion convention rather than the chat template.
Letting client code add special tokens when the serving layer already applies the template The server owns special tokens there, so any client-side BOS addition double-counts across every request.
Quick scan · 2–4 min
Everything important, quickly.
3 min: template already adds BOS + double BOS is out of distribution + position shift + attention anchoring + system-prompt/role-bleed symptoms + add_generation_prompt fix.
Code example
python
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-4-Maverick")
messages = [{"role": "system", "content": "Be terse."},
{"role": "user", "content": "Hi"}]
# WRONG: template adds BOS, then we add a second one.
ids = tok.apply_chat_template(messages, add_special_tokens=True)
ids = [tok.bos_token_id] + ids # double BOS, out of distribution
# RIGHT: let the template own all special tokens.
ids = tok.apply_chat_template(messages, add_generation_prompt=True)
assert ids.count(tok.bos_token_id) == 1 # guard against duplication
Real-world examples
Real products, models, and research that use this idea.
1HuggingFace Llama 4 and Mistral Large 3 checkpoints add BOS inside apply_chat_template, so a manual prepend reliably double-counts it.
2Teams fine-tuning open models routinely assert tokenizer output has exactly one BOS in tests, after debugging silent double-BOS quality regressions.
3vLLM and SGLang serving stacks let the chat template own special tokens precisely to avoid client-side BOS duplication across requests.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you write a regression test that catches double-BOS before it ships?
Sign in or go Pro to see the approach for this follow-up.
QWhy might a double EOS in the middle of a conversation be even worse than a double BOS?
Sign in or go Pro to see the approach for this follow-up.
QIf you fine-tune on data you tokenized with double BOS, what happens at inference with single BOS?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Prepending BOS manually after apply_chat_template already added it, creating a double-BOS sequence the model never saw in training.