Which token most often serves as an attention sink in pretrained autoregressive LLMs?

MCQ·Easy·4.0 · 0·~1 min·Asked atAdobe Freshworks Swiggy·Relevant atMeta

Attempt it

**The correct answer is C: the first token of the sequence, usually a BOS marker.** The mechanism is structural. Softmax forces every row of the attention weight matrix to sum to exactly 1. When a particular attention head has no content target that genuinely matters for the current query (which is true most of the time for most heads), the mandatory mass still has to go somewhere. The model learns to dump it at a fixed, content-light position. The first token is the natural choice for three reasons: - It is present in every sequence at a predictable index (position 0). - It is typically a special token like BOS whose embedding is content-light. - The post-attention contribution of sink mass becomes the BOS value vector, which is approximately zero in the residual stream. This pattern is called an attention sink. It was characterized by Xiao et al. in StreamingLLM (2023), which discovered that dropping the first few tokens from a sliding-window KV cache causes long-context generation to collapse. The fix is to keep the sinks permanently in the cache. **Why the other options are wrong.** Option A (recent tokens) is the locality bias, real but distinct from sink behavior. Option B (high-frequency words) is not what the empirical analysis shows. Option D (random) contradicts the strong consistent pattern across heads and models.

**The first token of the sequence, almost always the BOS marker.** Empirical work on pretrained transformer LMs consistently shows that many attention heads route a disproportionate share of their softmax mass to position 0, regardless of the query. **Why position 0 specifically.** Softmax forces each row of the attention weight matrix to sum to 1. For most heads, most of the time, there is no content match in the current context worth attending to. The mandatory probability mass has to land somewhere. The model converges during training to dump it at the most convenient position: - Present in every sequence the model sees (so the pattern can be learned consistently). - At a predictable, stable index (position 0). - With a content-light embedding (BOS is a special token whose value vector is approximately zero in the residual stream). No other position satisfies all three. Recent tokens shift each step. High-frequency words are not at predictable positions. Random positions are not learnable patterns. **Why this is not a bug.** The sink is the model's pragmatic solution to a structural problem: softmax has no zero vector, so a head that wants to output 'no key matters' has to fake it by routing mass to a content-light dump position. Sink behavior is observed in essentially every pretrained autoregressive LLM (Llama, Mistral, GPT-2 through GPT-4, Claude, Gemini). **Why it matters in practice.** StreamingLLM (Xiao et al. 2023) showed that sliding-window KV cache implementations that drop the sinks cause long-context generation to collapse within a few hundred tokens past the cache boundary. The standard fix is to keep the first 4-8 tokens permanently in the cache. Llama 4 Maverick, GPT-5.5, and most 2026 production stacks bake this in.

TL;DR

The very first token (typically BOS). It is present in every sequence at a predictable index and content-light, the ideal place to dump softmax's mandatory no-op mass.

Memory aid

Easy to grasp

Picture a classroom where every student is forced to raise a hand on every question, even when they do not know the answer. To keep things tidy, the class agrees that the kid in the front row is the official 'I do not know' designate, so anyone who is unsure points at them. Over the course of the year that front-row kid ends up with most of the don't-know votes, even on questions other kids actually want to answer. Transformer attention heads do the same thing with the first token of every sequence: it becomes the official 'I have nothing better to say' target.

Key concepts

Attention sink BOS token Softmax constraint StreamingLLM

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The first token, usually BOS, serves as the canonical attention sink in essentially every pretrained autoregressive LLM. This is a robust empirical finding across model families (GPT, Llama, Mistral, Claude, Gemini, DeepSeek), training data mixes, and tokenizer choices. The phenomenon is operator-level, not data-level: it emerges from the softmax probability constraint, and any position satisfying three structural criteria would serve as the sink. Position 0 wins because it is the unique position that satisfies all three.

This deep dive walks the structural argument, traces why each alternative option fails, explains the practical consequences for KV cache serving and quantization, and shows how modern fixes (register tokens, StreamingLLM, softmax-1) address the underlying constraint.

Mental model: the sink is not a bug in pretraining. It is the optimal solution to 'softmax has no zero vector but my head has no good content match'.

The three criteria a sink position must satisfy

For a position to serve as a learnable sink target, it must satisfy three jointly necessary criteria:

Universality

The position has to be present in every training sequence. If a candidate position is absent in some sequences, heads cannot learn to consistently dump mass there. Position 0 is the unique position guaranteed to exist in every non-empty sequence.

Position predictability

The position has to be at a stable, predictable index across sequences. Heads encode the dump target in their query-key inner product geometry, and that geometry has to point at the same absolute or relative position regardless of input. Position 0 is the cleanest stable position; recent positions shift every decoding step; high-frequency words like 'the' appear at random positions.

Content lightness

When sink mass w_sink multiplies the sink token's value vector v_sink, the contribution to the residual stream is w_sink * v_sink. If v_sink carries semantic content, that content gets injected into the residual stream every time a head dumps mass there, distorting downstream computation. BOS's embedding is content-light, so the injection is small.

Why position 0 wins

Only position 0 satisfies all three. Recent tokens fail predictability. High-frequency words fail predictability and content-lightness. Random positions fail predictability. BOS at position 0 wins by elimination.

Key insight: the choice of sink position is not arbitrary. It is the unique solution to the constraints, which is why every pretrained LLM converges to the same pattern.

Why softmax-must-sum-to-1 forces the sink to exist

Why the other options are wrong

Practical consequences and modern fixes

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Xiao et al. 2023 (StreamingLLM): characterized the sink phenomenon and showed that keeping the first 4 tokens in the KV cache enables million-token-scale streaming generation on Llama-2 7B.
Llama 4 Maverick, GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, DeepSeek V4: all exhibit classical BOS sinks; their production serving stacks include StreamingLLM-style sink retention.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does dropping BOS from a sliding-window KV cache cause long-context generation to collapse rather than degrade gracefully?

Every sink head depends on having BOS available to dump its mandatory mass. When BOS falls out of scope, those heads route their mass onto whatever happens to be at the start of the truncated window, a real content token. That token's representation gets corrupted by carrying both content signal and sink mass, the corruption propagates through the residual stream within a few layers, and downstream attention and MLP layers read corrupted inputs. The collapse is non-graceful because the failure is in the residual stream's geometry, not in any one attention pattern.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Picking 'most recent token' because recency biases attention. Recent tokens do receive a lot of attention, but not the sink-style no-op dump that defines attention sinks.

60 second bullets to scan on the way to the call.

Why position 0 is the natural sink and not any other position
How softmax's row sum to 1 forces mass to land somewhere

Primary sources. Browse if you want the original framing.