Find the bug: a Llama deployment serves traffic without prepending the BOS token.

Spot the error·Medium·4.0 · 0·~2 min

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

The bug is treating **BOS** (`<bos>`) as semantically empty. It is not. Llama and similar decoder-only models trained with BOS learn to use that position as an **attention sink**, a dump slot where softmax routes excess probability when no key strongly matches the query. Softmax must produce weights that sum to 1 across the keys. When no key is a strong match, the model has to put that weight somewhere. During training, it learns to dump that excess onto the BOS token, whose representation has no semantic content to protect. At inference, skip the BOS prepend and the first real prompt token inherits the sink role. That token now has two jobs: carry its actual semantic content and absorb softmax's leftover weight. The hidden state gets pulled in incompatible directions, and the model produces degraded outputs, especially in the first few generated tokens. Symptoms include weird openings, repetitive first sentences, off-topic completions, and a general quality drift compared to the same prompt with BOS prepended. The fix is one byte: always prepend the BOS token. This is one of the most common production bugs in homemade inference stacks. 'Tokenize and run' looks fine on toy inputs but degrades subtly under real traffic, and the symptom often gets misattributed to sampling, context length, or model selection.

Both claims invert the truth. BOS is not a no-op. It is the learned **attention sink** for Llama-style models. **Why softmax needs a sink.** Attention weights sum to exactly 1 per query, by softmax construction. When a query has no key it strongly wants to attend to, the model still has to put the unit of probability mass somewhere. During training, the model learns that dumping the excess on low-content positions is safe. The very first token (BOS) is the obvious candidate: no semantic content to protect, always at position 0. StreamingLLM formalized this and showed the first 4 tokens absorb most of the sink role, with BOS dominant. **What happens when BOS is missing.** The first real prompt token inherits the sink role. Its hidden state is pulled in two directions: encode its actual meaning, and serve as the dump slot for later queries without a strong match. The two roles are incompatible. The token's representation drifts off what training optimized for. **Symptoms.** Quality drop is most visible in the first 5 to 20 generated tokens. Openings become repetitive, sometimes off-topic, often a subtle quality drift that shallow QA misses. **Fix.** Always prepend the BOS token. Production tokenizer libraries expose `add_special_tokens=True` for this; custom pipelines must handle it explicitly. The fix is one byte; the diagnostic challenge is that the symptom is easy to misattribute to sampling or context length.

Quick reference

TL;DR

BOS acts as the learned attention sink. Drop it and the first real token inherits the sink role, distorting its hidden state and degrading the opening tokens of generation.

Memory aid

Easy to grasp

Imagine a sports team where one player is designated to catch every ball nobody else can field. Lose that player and someone else has to do their job on top of their own. The replacement player tries to play their normal position and catch random balls at the same time, doing both badly. The first token of every prompt (called BOS) is that designated catcher for the model. The system always has to put its leftover focus somewhere, and during training it learns to dump it on BOS because that slot carries no real content to protect. Skip BOS at runtime and the first real word of your prompt has to play two positions at once. Quality drops, especially on the first few words the model writes back.

Key concepts

BOS (Beginning-of-Sequence) token Attention sink StreamingLLM Tokenizer handling

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

6-8 min: softmax sum-to-1 invariant + attention sink phenomenon + BOS as learned sink + what happens when BOS is missing + symptoms in production + fix + connection to StreamingLLM.

Real-world examples

Real products, models, and research that use this idea.

Llama 2, 3, and 4 Maverick tokenizers all prepend BOS through Hugging Face's add_special_tokens flag.
The StreamingLLM paper (Xiao et al., MIT 2023) introduced the 'attention sink' terminology and showed the first 4 tokens carry the sink role.
Mistral 7B and Mistral Large require explicit BOS handling in some serving configurations; mishandled BOS is a documented production issue.
vLLM, TensorRT-LLM, and SGLang all handle BOS correctly through their tokenizer wrappers, while custom 'tokenize and run' stacks frequently miss it.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the attention sink concept connect to StreamingLLM?

QWhat if a model was trained without BOS?

QCan you diagnose missing BOS empirically?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating BOS as a no-op marker. It is a learned attention sink whose absence forces the first real token to absorb softmax's leftover weight, degrading the opening of generation.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.