Rescue a streaming deployment that collapses once early tokens leave the cache

Short answer·Hard·4.0 · 0·~3 min·Asked atAnyscale Coreweave Pwc

Attempt it

Your chat service streams unbounded conversations through a fixed 8k KV cache using a plain sliding window: oldest entries are evicted first. Quality is fine for the first 8k tokens, then output degenerates into repetition and gibberish, even though perplexity on 8k-token offline evals is excellent. Explain the architectural cause and design a fix that keeps memory bounded.

Free · 2 AI evals / day

**The architectural cause is the attention sink phenomenon.** Softmax sums to one. Each attention head MUST distribute its full attention budget across the keys it can see. Trained language models learn that, when no key has a strong relationship to the current query (which is most of the time for most heads), the safe move is to dump the excess mass on the earliest tokens, especially token 0, which is visible to every causal query in the sequence. Those early tokens become **attention sinks**: positions whose KV vectors got tuned during training to absorb miscellaneous attention mass without disrupting useful computation. A plain sliding window evicts the first tokens once the cache fills up. From the model's perspective, every head's softmax now redistributes over a key set that no longer includes the sinks it learned to lean on. The activations shift out of distribution across all layers. Generation degenerates. Offline evals do not catch this because they never evict; they always include the start-of-sequence tokens in their context. **The fix (StreamingLLM):** always pin the first 4 tokens in the cache and slide the window over everything after them.

**The cause: attention sinks.** Softmax forces each head to distribute its mass across visible keys, summing to exactly 1. For many heads on many queries, no key has a strong content match; the head still has to allocate the full budget. Trained models park excess mass on the earliest positions, especially token 0, because token 0 is visible to every causal query and the model tunes its K and V to absorb the dump without harming downstream computation. A plain sliding window evicts these sink tokens. Every head's softmax now renormalizes over a key set that excludes the sinks it learned to rely on. The mass redistributes to whichever remaining keys have next-largest scores, often the most-recent tokens. Activations shift out of distribution at every layer, gradients diverge from training, and output degenerates into repetition or gibberish. **Offline evals miss this** because they always include the start-of-sequence tokens. The bug appears only after the cache rolls over. **The fix is StreamingLLM** (Xiao et al. 2023). Maintain two cache regions: a small pinned prefix (typically 4 tokens) that stays forever, and a sliding window over the rest. Memory bounded at `sink_size + window_size`. The sinks remain visible to every query, the learned attention pattern continues to work, and streaming quality recovers to near-baseline. **Positional consistency.** Assign positions within the cache rather than by absolute sequence index. Sinks always sit at positions 0-3; window tokens occupy positions 4 to S+W-1.

TL;DR

Trained models park excess softmax mass on the first tokens (attention sinks); evicting them shifts every head's distribution out of distribution. Pin the first 4 tokens forever and slide the window over the rest.

Memory aid

Easy to grasp

Imagine a banker who must split exactly one dollar among customers every day, no more and no less. Some days only two customers show up but the banker still has to give the full dollar away. So the banker has a habit: any leftover money goes to the first customer of the day, every day, as a kind of overflow account. The customers and the banker have all gotten used to this. Now imagine a new policy that the first customer is sent home each morning. Suddenly the banker has no one to dump the leftover money on, the daily split is forced into shapes the banker has never seen, and decisions go sideways. Softmax has the same dump-the-leftover behavior; the first tokens are the overflow accounts. Evict them and everything breaks.

Key concepts

Transformer Architecture

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Streaming language model deployments are a problem where architecture meets engineering with a surprisingly subtle failure mode. The naive design is obvious: fix a KV cache size, evict oldest entries, generate forever. The reason it does not work is the attention sink phenomenon, a property of softmax-trained transformers that lets them silently rely on the earliest tokens of any sequence as a dumping ground for excess attention mass.

Xiao et al.'s StreamingLLM paper (2023) crystallized both the phenomenon and the fix: pin the first few tokens forever, slide the window over the rest. The result is bounded memory and stable quality at arbitrary conversation length. Subsequent work has explored training-time modifications that eliminate the need for the workaround, including dedicated sink tokens and softmax variants that permit attending to nothing.

This walkthrough explains why softmax forces the sink behavior, why eviction breaks the model's learned attention pattern, why offline evaluation cannot catch the bug, the specific mechanics of the StreamingLLM fix including the RoPE position-assignment detail, and the training-time alternatives that 2026 models are beginning to ship.

The softmax constraint and the attention sink

Softmax over K keys produces a strict probability distribution:

\text{softmax}(s)_i = \frac{e^{s_i}}{\sum_{j=1}^{K} e^{s_j}}, \quad \sum_i \text{softmax}(s)_i = 1

No zero option. No 'attend to nothing.' Every head, every query, every layer has to distribute exactly 1.0 of attention mass across the visible keys.

For any given query, most heads have nothing useful to attend to. The model trained these heads to specialize: one head looks for the previous-token name, another for syntactic agreement, a third for in-context retrieval triggers, and so on. On any given query the head's intended target either exists in context or it does not. When it does not, the head still has to allocate its full budget.

The trained solution is to dump the excess on the earliest positions. Reasons: (1) position 0 is visible to every causal query (the BOS token sees everything that comes after it), (2) the K and V of these early positions can be tuned during training to absorb mass without disrupting the rest of the computation, (3) the gradient pushes this from many heads simultaneously, so the early positions develop a robust sink role.

In a typical large LLM, you can verify the phenomenon by inspecting attention weights: many heads put 10-30% of their mass on position 0 across most queries, despite token 0 (often BOS) having no content meaningful for the query. The mass is going there because it has to go somewhere and position 0 is the safe dump.

This is what Xiao et al. termed the attention sink. It is not a bug; it is a learned strategy for handling the softmax budget constraint. The model trained this way and depends on it.

Why eviction breaks everything

The StreamingLLM fix in detail

Training-time alternatives and 2026 patterns

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Xiao et al.'s StreamingLLM paper (2023) is the canonical reference for the attention sink phenomenon and the pinned-prefix fix.
Evan Miller's 'Attention is off by one' blog post proposes the softmax modification that gives heads an 'attend to nothing' option.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you measure whether your specific model is sink-vulnerable before deploying StreamingLLM?

Compare attention weights at the BOS position across heads and layers. If many heads consistently put more than 5% mass on position 0 regardless of query, you have heavy sink use; the model is vulnerable.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Blaming the model's tokenizer, the RoPE base, or the sliding-window kernel implementation. The cause is the softmax sum-to-one constraint interacting with the model's learned habit of using early tokens as overflow.

60 second bullets to scan on the way to the call.

Why softmax's sum-to-one constraint forces excess mass somewhere
How trained models learn to use early tokens as sinks

Primary sources. Browse if you want the original framing.