Find the bug: a Llama deployment serves traffic without prepending the BOS token.

Spot the error·Medium·4.0 · 0·~2 min·Asked atOracle Robinhood Samsung·Relevant atAi4bharat Cerebras Deepseek Microsoft

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

The bug is treating **BOS** (`<bos>`) as semantically empty. It is not. Llama and similar decoder-only models trained with BOS learn to use that position as an **attention sink**, a dump slot where softmax routes excess probability when no key strongly matches the query. Softmax must produce weights that sum to 1 across the keys. When no key is a strong match, the model has to put that weight somewhere. During training, it learns to dump that excess onto the BOS token, whose representation has no semantic content to protect. At inference, skip the BOS prepend and the first real prompt token inherits the sink role. That token now has two jobs: carry its actual semantic content and absorb softmax's leftover weight. The hidden state gets pulled in incompatible directions, and the model produces degraded outputs, especially in the first few generated tokens. Symptoms include weird openings, repetitive first sentences, off-topic completions, and a general quality drift compared to the same prompt with BOS prepended. The fix is one byte: always prepend the BOS token. This is one of the most common production bugs in homemade inference stacks. 'Tokenize and run' looks fine on toy inputs but degrades subtly under real traffic, and the symptom often gets misattributed to sampling, context length, or model selection.

Both claims invert the truth. BOS is not a no-op. It is the learned **attention sink** for Llama-style models. **Why softmax needs a sink.** Attention weights sum to exactly 1 per query, by softmax construction. When a query has no key it strongly wants to attend to, the model still has to put the unit of probability mass somewhere. During training, the model learns that dumping the excess on low-content positions is safe. The very first token (BOS) is the obvious candidate: no semantic content to protect, always at position 0. StreamingLLM formalized this and showed the first 4 tokens absorb most of the sink role, with BOS dominant. **What happens when BOS is missing.** The first real prompt token inherits the sink role. Its hidden state is pulled in two directions: encode its actual meaning, and serve as the dump slot for later queries without a strong match. The two roles are incompatible. The token's representation drifts off what training optimized for. **Symptoms.** Quality drop is most visible in the first 5 to 20 generated tokens. Openings become repetitive, sometimes off-topic, often a subtle quality drift that shallow QA misses. **Fix.** Always prepend the BOS token. Production tokenizer libraries expose `add_special_tokens=True` for this; custom pipelines must handle it explicitly. The fix is one byte; the diagnostic challenge is that the symptom is easy to misattribute to sampling or context length.

TL;DR

BOS acts as the learned attention sink. Drop it and the first real token inherits the sink role, distorting its hidden state and degrading the opening tokens of generation.

Memory aid

Easy to grasp

Imagine a sports team where one player is designated to catch every ball nobody else can field. Lose that player and someone else has to do their job on top of their own. The replacement player tries to play their normal position and catch random balls at the same time, doing both badly. The first token of every prompt (called BOS) is that designated catcher for the model. The system always has to put its leftover focus somewhere, and during training it learns to dump it on BOS because that slot carries no real content to protect. Skip BOS at runtime and the first real word of your prompt has to play two positions at once. Quality drops, especially on the first few words the model writes back.

Key concepts

BOS (Beginning-of-Sequence) token Attention sink StreamingLLM Tokenizer handling

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The BOS token looks like nothing, just a single special token at the start of the sequence. In reality, for Llama-family and most decoder-only models trained with it, BOS plays a load-bearing architectural role: it is the learned attention sink, the dump slot where softmax routes excess probability when no key strongly matches the query.

This deep dive walks through the softmax invariant that forces a sink to exist, explains why BOS becomes the sink during training, traces what happens at inference when BOS is missing, lists the symptoms, and connects to the StreamingLLM line of work that formalized the attention sink phenomenon.

Mental model: softmax must produce weights summing to 1. The model learns to dump leftover probability somewhere safe. BOS, with no semantic content to protect, is the obvious safe slot.

Why an attention sink must exist

The softmax invariant

For any query, the attention weights across keys sum to exactly 1. This is the defining property of softmax.

The implication

When the query is early in the sequence (few keys to attend to), or when no key is a strong semantic match, the model still has to assign the full unit of probability somewhere. There is no 'attend to nothing' option, attention weights cannot all be small if they must sum to 1.

The dump-slot problem

If the model dumps that leftover probability onto a semantically meaningful key, the context vector gets polluted with that key's value vector. The query's downstream representation drifts off the intended computation.

The solution training discovers: dump it onto a position whose value vector is uninformative. The first token of every training sequence is the natural candidate.

Why this is universal

Every softmax-based attention layer has the same invariant. Empirically, every decoder-only LLM trained on natural language data learns some form of attention sink. The difference is which token absorbs the sink role, BOS if the model was trained with it, or the first content token if not.

How BOS becomes the learned sink

What happens when BOS is missing at inference

Fixing it and avoiding related traps

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Llama 2, 3, and 4 Maverick tokenizers all prepend BOS through Hugging Face's add_special_tokens flag.
The StreamingLLM paper (Xiao et al., MIT 2023) introduced the 'attention sink' terminology and showed the first 4 tokens carry the sink role.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the attention sink concept connect to StreamingLLM?

StreamingLLM proposed keeping the first 4 tokens permanently in the KV cache plus a sliding window, because those first 4 tokens carry the attention sink role. Drop them via naive sliding and quality collapses; keep them and you can stream indefinitely. BOS is the dominant sink token among those first 4.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating BOS as a no-op marker. It is a learned attention sink whose absence forces the first real token to absorb softmax's leftover weight, degrading the opening of generation.

60 second bullets to scan on the way to the call.

What an attention sink is and why it exists
Why softmax forces leftover probability to be assigned somewhere

Primary sources. Browse if you want the original framing.