**The correct answer is C: the first token of the sequence, usually a BOS marker.**
The mechanism is structural. Softmax forces every row of the attention weight matrix to sum to exactly 1. When a particular attention head has no content target that genuinely matters for the current query (which is true most of the time for most heads), the mandatory mass still has to go somewhere. The model learns to dump it at a fixed, content-light position.
The first token is the natural choice for three reasons:
- It is present in every sequence at a predictable index (position 0).
- It is typically a special token like BOS whose embedding is content-light.
- The post-attention contribution of sink mass becomes the BOS value vector, which is approximately zero in the residual stream.
This pattern is called an attention sink. It was characterized by Xiao et al. in StreamingLLM (2023), which discovered that dropping the first few tokens from a sliding-window KV cache causes long-context generation to collapse. The fix is to keep the sinks permanently in the cache.
**Why the other options are wrong.** Option A (recent tokens) is the locality bias, real but distinct from sink behavior. Option B (high-frequency words) is not what the empirical analysis shows. Option D (random) contradicts the strong consistent pattern across heads and models.
**The first token of the sequence, almost always the BOS marker.** Empirical work on pretrained transformer LMs consistently shows that many attention heads route a disproportionate share of their softmax mass to position 0, regardless of the query.
**Why position 0 specifically.** Softmax forces each row of the attention weight matrix to sum to 1. For most heads, most of the time, there is no content match in the current context worth attending to. The mandatory probability mass has to land somewhere. The model converges during training to dump it at the most convenient position:
- Present in every sequence the model sees (so the pattern can be learned consistently).
- At a predictable, stable index (position 0).
- With a content-light embedding (BOS is a special token whose value vector is approximately zero in the residual stream).
No other position satisfies all three. Recent tokens shift each step. High-frequency words are not at predictable positions. Random positions are not learnable patterns.
**Why this is not a bug.** The sink is the model's pragmatic solution to a structural problem: softmax has no zero vector, so a head that wants to output 'no key matters' has to fake it by routing mass to a content-light dump position. Sink behavior is observed in essentially every pretrained autoregressive LLM (Llama, Mistral, GPT-2 through GPT-4, Claude, Gemini).
**Why it matters in practice.** StreamingLLM (Xiao et al. 2023) showed that sliding-window KV cache implementations that drop the sinks cause long-context generation to collapse within a few hundred tokens past the cache boundary. The standard fix is to keep the first 4-8 tokens permanently in the cache. Llama 4 Maverick, GPT-5.5, and most 2026 production stacks bake this in.
Quick reference
TL;DR
The very first token (typically BOS). It is present in every sequence at a predictable index and content-light, the ideal place to dump softmax's mandatory no-op mass.
Picture a classroom where every student is forced to raise a hand on every question, even when they do not know the answer. To keep things tidy, the class agrees that the kid in the front row is the official 'I do not know' designate, so anyone who is unsure points at them. Over the course of the year that front-row kid ends up with most of the don't-know votes, even on questions other kids actually want to answer. Transformer attention heads do the same thing with the first token of every sequence: it becomes the official 'I have nothing better to say' target.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Models trained with register tokens (Darcet et al. style) Registers are the dedicated no-op slots, so heads route mass to registers rather than the first content position. Sink behavior on BOS shrinks or disappears entirely.
Models using softmax-1 or attention normalization variants that allow rows to sum to < 1 The constraint that creates sinks is relaxed, so the model has no need to route excess mass to a fixed dump position.
Encoder-only models like BERT during MLM pretraining Bidirectional attention plus the CLS token convention changes the sink structure; analysis usually attributes special-token behavior to CLS rather than to a generic 'first content token' sink.
Linear or kernel-attention models like Performer The softmax row-sum constraint applies differently after feature-map factorization, and the classical sink pattern does not arise in the same way.
Quick scan · 2–4 min
Everything important, quickly.
Name BOS as the canonical sink, derive why position 0 is the unique convergent dump target, link to StreamingLLM's sliding-window fix, and connect to the outlier-activation quantization problem.
Real-world examples
Real products, models, and research that use this idea.
1Xiao et al. 2023 (StreamingLLM): characterized the sink phenomenon and showed that keeping the first 4 tokens in the KV cache enables million-token-scale streaming generation on Llama-2 7B.
2Llama 4 Maverick, GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, DeepSeek V4: all exhibit classical BOS sinks; their production serving stacks include StreamingLLM-style sink retention.
3Dettmers et al. 2022 (LLM.int8): identified the outlier-activation problem at sink-bearing positions, motivating mixed-precision quantization schemes.
4Darcet et al. 2024 (Vision Transformers Need Registers): showed the same sink phenomenon in ViT and demonstrated that register tokens eliminate it.
5Mistral 7B and Mixtral exhibit BOS sinks; sliding-window attention serving requires sink retention to avoid degraded long-context behavior.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does dropping BOS from a sliding-window KV cache cause long-context generation to collapse rather than degrade gracefully?
Sign in or go Pro to see the approach for this follow-up.
QHow does the attention-sink phenomenon connect to the difficulty of quantizing LLMs to int8 or int4?
Sign in or go Pro to see the approach for this follow-up.
QWhat changes if a model has multiple sink positions instead of just position 0?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Picking 'most recent token' because recency biases attention. Recent tokens do receive a lot of attention, but not the sink-style no-op dump that defines attention sinks.