Click any words you think contain an error. Click again to unmark.
Mark at least one word to submit.
The bug is treating **BOS** (`<bos>`) as semantically empty. It is not. Llama and similar decoder-only models trained with BOS learn to use that position as an **attention sink**, a dump slot where softmax routes excess probability when no key strongly matches the query.
Softmax must produce weights that sum to 1 across the keys. When no key is a strong match, the model has to put that weight somewhere. During training, it learns to dump that excess onto the BOS token, whose representation has no semantic content to protect.
At inference, skip the BOS prepend and the first real prompt token inherits the sink role. That token now has two jobs: carry its actual semantic content and absorb softmax's leftover weight. The hidden state gets pulled in incompatible directions, and the model produces degraded outputs, especially in the first few generated tokens.
Symptoms include weird openings, repetitive first sentences, off-topic completions, and a general quality drift compared to the same prompt with BOS prepended. The fix is one byte: always prepend the BOS token.
This is one of the most common production bugs in homemade inference stacks. 'Tokenize and run' looks fine on toy inputs but degrades subtly under real traffic, and the symptom often gets misattributed to sampling, context length, or model selection.
Both claims invert the truth. BOS is not a no-op. It is the learned **attention sink** for Llama-style models.
**Why softmax needs a sink.** Attention weights sum to exactly 1 per query, by softmax construction. When a query has no key it strongly wants to attend to, the model still has to put the unit of probability mass somewhere.
During training, the model learns that dumping the excess on low-content positions is safe. The very first token (BOS) is the obvious candidate: no semantic content to protect, always at position 0. StreamingLLM formalized this and showed the first 4 tokens absorb most of the sink role, with BOS dominant.
**What happens when BOS is missing.** The first real prompt token inherits the sink role. Its hidden state is pulled in two directions: encode its actual meaning, and serve as the dump slot for later queries without a strong match. The two roles are incompatible. The token's representation drifts off what training optimized for.
**Symptoms.** Quality drop is most visible in the first 5 to 20 generated tokens. Openings become repetitive, sometimes off-topic, often a subtle quality drift that shallow QA misses.
**Fix.** Always prepend the BOS token. Production tokenizer libraries expose `add_special_tokens=True` for this; custom pipelines must handle it explicitly. The fix is one byte; the diagnostic challenge is that the symptom is easy to misattribute to sampling or context length.
Quick reference
TL;DR
BOS acts as the learned attention sink. Drop it and the first real token inherits the sink role, distorting its hidden state and degrading the opening tokens of generation.
Imagine a sports team where one player is designated to catch every ball nobody else can field. Lose that player and someone else has to do their job on top of their own. The replacement player tries to play their normal position and catch random balls at the same time, doing both badly. The first token of every prompt (called BOS) is that designated catcher for the model. The system always has to put its leftover focus somewhere, and during training it learns to dump it on BOS because that slot carries no real content to protect. Skip BOS at runtime and the first real word of your prompt has to play two positions at once. Quality drops, especially on the first few words the model writes back.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Embedding models or encoder-only models that were not trained with BOS as a sink token The attention sink role is specific to decoder-only models trained with a leading sentinel; encoder-only architectures may have different conventions or no sink at all.
Multi-turn chat where BOS is already embedded in the chat template Templates like Llama 3 chat or Mistral instruct put BOS at the right place automatically; manually adding another BOS creates an out of distribution input.
Mid-stream continuation of an already-generated context BOS only goes at the very start of a sequence; injecting it mid-stream would create a discontinuity that the model never saw during training.
Quick scan · 2–4 min
Everything important, quickly.
6-8 min: softmax sum-to-1 invariant + attention sink phenomenon + BOS as learned sink + what happens when BOS is missing + symptoms in production + fix + connection to StreamingLLM.
Real-world examples
Real products, models, and research that use this idea.
1Llama 2, 3, and 4 Maverick tokenizers all prepend BOS through Hugging Face's add_special_tokens flag.
2The StreamingLLM paper (Xiao et al., MIT 2023) introduced the 'attention sink' terminology and showed the first 4 tokens carry the sink role.
3Mistral 7B and Mistral Large require explicit BOS handling in some serving configurations; mishandled BOS is a documented production issue.
4vLLM, TensorRT-LLM, and SGLang all handle BOS correctly through their tokenizer wrappers, while custom 'tokenize and run' stacks frequently miss it.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the attention sink concept connect to StreamingLLM?
Sign in or go Pro to see the approach for this follow-up.
QWhat if a model was trained without BOS?
Sign in or go Pro to see the approach for this follow-up.
QCan you diagnose missing BOS empirically?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating BOS as a no-op marker. It is a learned attention sink whose absence forces the first real token to absorb softmax's leftover weight, degrading the opening of generation.