Sliding-window attention (used in Mistral, Longformer, etc.) restricts each token to attending only to the previous W tokens. Naively this seems to throw away anything older than W positions back. Describe the architectural property that lets information from older tokens still reach the current position, and explain how the effective receptive field scales.
Free · 2 AI evals / day
The trick is **layer stacking**, and the carrier is the residual stream.
At layer 1, position `t` directly attends to positions in `[t - W + 1, t]`, the literal window. So far so naive: no token older than `t - W + 1` is visible.
At layer 2, position `t` again attends to its window, the layer-1 hidden states at positions `[t - W + 1, t]`. But each of those layer-1 hidden states is itself a function of ITS window, which extends one further window back. So through one hop, layer 2 at position `t` can be influenced by anything in `[t - 2W + 2, t]`.
After `L` layers, the effective receptive field is roughly `W * L` positions. For Mistral-7B with `W = 4096` and `L = 32`, that is `4096 * 32 = 131,072` tokens, well past the nominal 8k context length. The model can be influenced by content from arbitrarily far back, just indirectly through depth-hops.
The residual stream is what makes this work cleanly. Each layer's attention output is ADDED to the residual stream, not replacing it, so information that propagated forward at layer 5 is still available at layer 12 without needing a fresh attention edge.
**The catch:** the long-range signal is indirect and bandwidth-limited. Each hop has to repack the relevant info into a single hidden state, and the model has to learn what is worth carrying forward.
**Layer stacking is the mechanism. Effective receptive field grows linearly with depth: roughly `W * L`.**
At a single layer, sliding-window attention restricts each token to a window of `W` keys behind it. Layer 1 at position `t` can attend to `[t - W + 1, t]`, only the literal window.
The key insight is what happens at layer 2. The hidden state at position `t - W + 1` in layer 1 is already a function of layer-0 positions in `[t - 2W + 2, t - W + 1]`, because that token's own layer-1 attention used a window of W keys behind it. So when layer-2 attention at position `t` reads the layer-1 hidden state at position `t - W + 1`, it indirectly absorbs information from positions up to `t - 2W + 2`.
Iterating, after `L` layers the effective receptive field is approximately `W * L` positions.
**Concrete numbers for Mistral 7B:**
- Window `W = 4096`
- Depth `L = 32`
- Effective receptive field `≈ 4096 * 32 = 131,072 tokens`
That is 16x larger than the model's nominal 8k pretraining context length. The model can absorb information from anywhere in a 130k-token input, despite no single attention layer ever looking past 4k.
**The residual stream is the carrier.** Every layer's attention output is added to the residual stream rather than overwriting it, so once information has propagated forward at layer `k`, it remains available in the residual at all later layers without needing a fresh attention edge.
**The trade-off versus full attention.** Sliding-window with depth-hops is far cheaper, `O(W * T * L)` instead of `O(T^2 * L)`, but the long-range signal is indirect and bandwidth-limited: each hop has to repack relevant info into a single hidden state. Full attention can route long-range information in one shot and remains more expressive when affordable.
Quick reference
TL;DR
Layer stacking. Each layer adds W tokens of reach; after L layers the effective receptive field is roughly W * L. Mistral-7B's 4k window over 32 layers gives ~131k token reach.
Picture a line of telephone-game players where each player can only whisper to the four people immediately in front of them. A message from the back of the line cannot reach the front in one hop, but as the line passes the message forward four people at a time, it eventually arrives at the front. The total distance the message can travel is four people per hop times however many hops there are. Sliding-window attention works the same way: each layer is one hop, the window size is how many tokens you can reach per hop, and the depth of the transformer is how many hops the message gets.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Long-context tasks requiring direct retrieval of specific tokens from far back in the context (needle in haystack) Multi-hop propagation through depth bandwidth-limits the signal, so direct full attention or a hybrid with periodic full-attention layers performs better on retrieval-heavy tasks.
Short-context inference workloads where T < W anyway When the input length is already smaller than the sliding window, the windowing is a no-op and full attention is mathematically equivalent at the same cost.
Training stacks that can afford full attention at the target context length Full attention with FlashAttention v2 or v3 is competitive on training cost at context lengths up to ~16-32k, and avoids the depth-hop bandwidth bottleneck. Llama 3 moved this direction.
Very shallow models (L < 8) With few layers, W * L is small enough that the effective receptive field cannot reach the full input length, regardless of how the windowing is structured.
Quick scan · 2–4 min
Everything important, quickly.
Explain that per-layer window W combines with depth L through the residual stream to give a receptive field of roughly W * L, run the Mistral 7B numbers (4096 * 32 = 131k), note the bandwidth-bottleneck caveat, and contrast with full attention and hybrid long-context architectures.
Side-by-side
Attention pattern
Per-layer cost
Per-layer reach
Effective receptive field
Full attention
O(T^2 * d_k)
T (everything)
T, direct in one hop
Sliding window
O(W * T * d_k)
W tokens back
≈ W * L via depth-hops
SSM / Mamba
O(T * d_state)
Recurrent, all of history compressed
Bounded by d_state bandwidth
Hybrid (sliding + periodic full)
Mostly O(W * T) + sparse O(T^2)
W per layer, plus full-attention layers
≈ W * L with one-hop long edges at full-attn layers
Real-world examples
Real products, models, and research that use this idea.
1Mistral 7B: W = 4096, L = 32, theoretical receptive field ≈ 131k tokens despite nominal 8k pretraining context.
2Longformer (Beltagy et al. 2020): pioneered sliding-window attention with W = 512 plus global tokens for document-level tasks.
3Gemini 3.1 Pro: uses a hybrid attention stack with periodic full-attention layers interleaved with sliding-window layers to balance cost and long-range routing.
4Jamba (AI21 2024): alternates Mamba SSM layers with sliding-window attention layers, exploiting both depth-hops and recurrent state for million-token contexts.
5Llama 3 and Llama 4: deliberately avoid sliding-window in favor of full attention with longer pretraining contexts, trading compute for direct long-range routing.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the effective receptive field of a sliding-window model smaller than the theoretical W * L?
Sign in or go Pro to see the approach for this follow-up.
QHow does the residual stream's add-not-replace structure interact with sliding-window attention?
Sign in or go Pro to see the approach for this follow-up.
QWhy do modern long-context stacks like Gemini 3.1 Pro and Jamba interleave sliding-window with full attention or SSM layers?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming sliding-window models cannot see past W tokens. The per-layer window is W; the cross-layer receptive field is L * W via stacking.