Which token most often serves as an attention sink in pretrained autoregressive LLMs?

**The correct answer is C: the first token of the sequence, usually a BOS marker.** The mechanism is structural. Softmax forces every row of the attention weight matrix to sum to exactly 1. When a particular attention head has no content target that genuinely matters for the current query (which is true most of the time for most heads), the mandatory mass still has to go somewhere. The model learns to dump it at a fixed, content-light position. The first token is the natural choice for three reasons: - It is present in every sequence at a predictable index (position 0). - It is typically a special token like BOS whose embedding is content-light. - The post-attention contribution of sink mass becomes the BOS value vector, which is approximately zero in the residual stream. This pattern is called an attention sink. It was characterized by Xiao et al. in StreamingLLM (2023), which discovered that dropping the first few tokens from a sliding-window KV cache causes long-context generation to collapse. The fix is to keep the sinks permanently in the cache. **Why the other options are wrong.** Option A (recent tokens) is the locality bias, real but distinct from sink behavior. Option B (high-frequency words) is not what the empirical analysis shows. Option D (random) contradicts the strong consistent pattern across heads and models.

**The first token of the sequence, almost always the BOS marker.** Empirical work on pretrained transformer LMs consistently shows that many attention heads route a disproportionate share of their softmax mass to position 0, regardless of the query. **Why position 0 specifically.** Softmax forces each row of the attention weight matrix to sum to 1. For most heads, most of the time, there is no content match in the current context worth attending to. The mandatory probability mass has to land somewhere. The model converges during training to dump it at the most convenient position: - Present in every sequence the model sees (so the pattern can be learned consistently). - At a predictable, stable index (position 0). - With a content-light embedding (BOS is a special token whose value vector is approximately zero in the residual stream). No other position satisfies all three. Recent tokens shift each step. High-frequency words are not at predictable positions. Random positions are not learnable patterns. **Why this is not a bug.** The sink is the model's pragmatic solution to a structural problem: softmax has no zero vector, so a head that wants to output 'no key matters' has to fake it by routing mass to a content-light dump position. Sink behavior is observed in essentially every pretrained autoregressive LLM (Llama, Mistral, GPT-2 through GPT-4, Claude, Gemini). **Why it matters in practice.** StreamingLLM (Xiao et al. 2023) showed that sliding-window KV cache implementations that drop the sinks cause long-context generation to collapse within a few hundred tokens past the cache boundary. The standard fix is to keep the first 4-8 tokens permanently in the cache. Llama 4 Maverick, GPT-5.5, and most 2026 production stacks bake this in.