Reason about KV cache growth in a sliding window model pushed to 32k tokens

MCQ·Medium·4.0 · 0·~1 min·Asked atBaseten Jump Trading Robust Intelligence

Attempt it

Sliding-window attention restricts each layer's attention pattern: token `i` can only attend to the previous `w` tokens (e.g. `w = 4096`), not the full history. Two consequences follow. **Consequence 1: bounded KV cache per layer.** Because a layer never reads beyond its window, it never needs to store more than `w` entries in its KV cache. The cache becomes a rolling buffer of size `w`: when a new token arrives, the oldest entry can be evicted. At 32k context with a 4k window, each layer's cache is capped at 4096 entries instead of 32000. **Consequence 2: the model is NOT limited to a 4k effective context.** This is the counterintuitive part. Tokens beyond 4096 positions back are not directly attended to by any layer, but they CAN influence later predictions. The mechanism is the stack itself: at layer `k`, the attention reads from layer `k-1`'s output, which was computed using layer `k-2`'s output, which had its own window over earlier tokens, and so on. Information about token at position `t` can hop forward through layers and reach positions `t + L * w` away, where `L` is depth.

The correct answer is **A**. Sliding-window attention caps each layer's KV cache at the window size, but stacking layers extends the effective receptive field. **KV cache math.** A full-attention layer at 32k context stores `32000` entries per layer. A sliding-window layer with `w = 4096` stores at most `4096` entries per layer. At L=32 layers, the cache shrinks from `32 * 32000 = 1.02M` entries to `32 * 4096 = 131k` entries: an 8x reduction. For a Mistral 7B style model at d_model=4096, n_kv_heads=8, d_head=128, that is roughly `32 * 4096 * 8 * 128 * 2 bytes = 268 MB` per sequence, versus `2.1 GB` at full attention. **Effective receptive field grows with depth.** Layer 1 sees tokens `[i-w, i]`. Layer 2 reads layer-1 outputs, which themselves encode the previous window's content, so layer 2 effectively sees information from `[i-2w, i]`. Generalising, layer `k` has effective receptive field `[i - kw, i]`. For Mistral with `w = 4096, L = 32`, the theoretical reach is around 131k tokens. **Why the reach is fuzzy.** Each layer hop compresses information; the signal from a token 100k positions ago is not as faithful as direct attention. Long-range recall benchmarks (needle in a haystack) show sliding-window models degrade gracefully but not as crisply as full-attention models. This is why Gemma-class models interleave a small number of global layers to restore exact long-range access. **Why the wrong options are wrong.** Option B confuses window with batched compute; the cache truly is bounded. Option C ignores the layer-to-layer relay. Option D invents a query cache that does not exist (queries are recomputed per step).

TL;DR

Each sliding-window layer caps its KV cache at the window size (e.g. 4096), but stacking layers extends the effective receptive field to roughly window times depth.

Memory aid

Easy to grasp

Picture each transformer layer as a person standing in line, allowed to look only at the 4096 people immediately in front of them. That person summarises what they see and whispers it forward to the next layer's person, who also looks only at their own 4096 neighbours. Even though no single person sees beyond 4096, the whispered summaries hop forward, so by layer 20 the whisper carries information from far further back. Storage per person is bounded at 4096, which is exactly what makes the design memory-efficient at long context. The illusion of a longer reach comes from stacking, not from anyone holding more cache.

Key concepts

Transformer Architecture

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Sliding-window attention is the most studied of the production-grade approximations to full attention. Mistral 7B's success made it a standard tool in the LLM architect's kit, and the question 'what does the cache look like at long context' is the right way to test whether someone understands the design or just knows the name.

This walkthrough explains the bounded-cache property, the depth-mediated reach, the practical limits of multi-hop information flow, and why hybrid designs that interleave a few global layers emerged.

Mental model: sliding window bounds the per-layer cache to w entries. Depth extends the effective receptive field to ~L*w. The first effect is exact; the second effect is real but degrades with hops.

Per-layer cache as a rolling buffer

The bounded-cache property

In a sliding-window layer with window w, token at position i attends only to positions [max(0, i-w+1), i]. The layer's KV cache therefore never needs to hold more than w entries per attention head: anything older is provably unread.

Implementations treat the cache as a circular buffer:

code

cache[head, slot] for slot in 0..w-1

When a new token arrives at position i > w, the entry for position i - w is overwritten. The total memory footprint is independent of context length.

Memory savings at scale

For Mistral 7B at 32k context: full attention would need 32 layers * 32000 tokens * 8 KV heads * 128 d_head * 2 bytes = 2.1 GB per sequence. Sliding-window with w=4096 needs 32 * 4096 * 8 * 128 * 2 = 268 MB per sequence. An 8x reduction, which is the difference between fitting many concurrent sequences on one GPU and only a few.

Compute savings too

Attention compute also scales with min(w, T). At long context, this is the difference between O(T w) (sliding) and O(T^2) (full). At 32k context with w=4096, full attention is 8x more compute per layer.

Depth-mediated reach: how stacking saves us

Hybrid designs: sliding plus a few global

Implementation details and gotchas

Putting numbers to it: 2026 frontier-model context

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Mistral 7B pioneered the production sliding-window design with w=4096 and 32 layers, theoretical reach ~131k.
Mixtral 8x7B keeps the same sliding-window-only attention as Mistral 7B in each expert.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the KV cache memory scale exactly under sliding-window attention?

Per sequence: L * w * n_kv_heads * d_head * 2 bytes (bf16/fp16). Independent of context length once context exceeds w. This is the production win at long context.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Believing tokens past the window are unreachable. They are reachable indirectly via layer-to-layer relay, which is the whole point of stacking sliding-window layers.

60 second bullets to scan on the way to the call.

Why a sliding-window layer's KV cache stays bounded at w
How depth extends the effective receptive field to O(L w)

Primary sources. Browse if you want the original framing.