Sliding window attention can't see beyond the window. Describe the trick that recovers older context.

Short answer·Medium·4.0 · 0·~3 min·Asked atGroq Robinhood Weaviate·Relevant atMistral AI

Attempt it

Sliding window attention (used in Mistral, Longformer, etc.) restricts each token to attending only to the previous W tokens. Naively this seems to throw away anything older than W positions back. Describe the architectural property that lets information from older tokens still reach the current position, and explain how the effective receptive field scales.

Free · 2 AI evals / day

The trick is **layer stacking**, and the carrier is the residual stream. At layer 1, position `t` directly attends to positions in `[t - W + 1, t]`, the literal window. So far so naive: no token older than `t - W + 1` is visible. At layer 2, position `t` again attends to its window, the layer-1 hidden states at positions `[t - W + 1, t]`. But each of those layer-1 hidden states is itself a function of ITS window, which extends one further window back. So through one hop, layer 2 at position `t` can be influenced by anything in `[t - 2W + 2, t]`. After `L` layers, the effective receptive field is roughly `W * L` positions. For Mistral-7B with `W = 4096` and `L = 32`, that is `4096 * 32 = 131,072` tokens, well past the nominal 8k context length. The model can be influenced by content from arbitrarily far back, just indirectly through depth-hops. The residual stream is what makes this work cleanly. Each layer's attention output is ADDED to the residual stream, not replacing it, so information that propagated forward at layer 5 is still available at layer 12 without needing a fresh attention edge. **The catch:** the long-range signal is indirect and bandwidth-limited. Each hop has to repack the relevant info into a single hidden state, and the model has to learn what is worth carrying forward.

**Layer stacking is the mechanism. Effective receptive field grows linearly with depth: roughly `W * L`.** At a single layer, sliding-window attention restricts each token to a window of `W` keys behind it. Layer 1 at position `t` can attend to `[t - W + 1, t]`, only the literal window. The key insight is what happens at layer 2. The hidden state at position `t - W + 1` in layer 1 is already a function of layer-0 positions in `[t - 2W + 2, t - W + 1]`, because that token's own layer-1 attention used a window of W keys behind it. So when layer-2 attention at position `t` reads the layer-1 hidden state at position `t - W + 1`, it indirectly absorbs information from positions up to `t - 2W + 2`. Iterating, after `L` layers the effective receptive field is approximately `W * L` positions. **Concrete numbers for Mistral 7B:** - Window `W = 4096` - Depth `L = 32` - Effective receptive field `≈ 4096 * 32 = 131,072 tokens` That is 16x larger than the model's nominal 8k pretraining context length. The model can absorb information from anywhere in a 130k-token input, despite no single attention layer ever looking past 4k. **The residual stream is the carrier.** Every layer's attention output is added to the residual stream rather than overwriting it, so once information has propagated forward at layer `k`, it remains available in the residual at all later layers without needing a fresh attention edge. **The trade-off versus full attention.** Sliding-window with depth-hops is far cheaper, `O(W * T * L)` instead of `O(T^2 * L)`, but the long-range signal is indirect and bandwidth-limited: each hop has to repack relevant info into a single hidden state. Full attention can route long-range information in one shot and remains more expressive when affordable.

TL;DR

Layer stacking. Each layer adds W tokens of reach; after L layers the effective receptive field is roughly W * L. Mistral-7B's 4k window over 32 layers gives ~131k token reach.

Memory aid

Easy to grasp

Picture a line of telephone-game players where each player can only whisper to the four people immediately in front of them. A message from the back of the line cannot reach the front in one hop, but as the line passes the message forward four people at a time, it eventually arrives at the front. The total distance the message can travel is four people per hop times however many hops there are. Sliding-window attention works the same way: each layer is one hop, the window size is how many tokens you can reach per hop, and the depth of the transformer is how many hops the message gets.

Key concepts

Sliding-window attention Receptive field Residual stream Layer stacking

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Sliding-window attention restricts each layer to a window of W keys behind each query, but the residual stream and layer stacking together extend the effective receptive field to roughly W * L. That is the canonical answer, and Mistral 7B is the standard example: W = 4096 across 32 layers gives a theoretical receptive field of ≈ 131k tokens, despite the model's nominal 8k pretraining context.

This deep dive walks the mechanism step by step, derives the W * L scaling, runs concrete numbers for production models, explains why the theoretical bound is not the effective bound, and shows where sliding-window fits in the broader 2026 long-context design space alongside full attention, SSMs, and hybrid architectures.

Mental model: per layer W, per stack W * L. The residual stream is the highway that carries depth-hops without re-attention.

The per-layer and cross-layer reach

Sliding-window attention is a structural restriction on the attention mask: each query position can attend only to the W most recent key positions.

One-layer reach

At layer 1, position t can attend to positions in [t - W + 1, t]. Anything older than t - W + 1 is invisible at this layer.

Two-layer reach

At layer 2, position t again attends to [t - W + 1, t] in terms of layer-1 hidden states. But each of those layer-1 hidden states is itself a function of ITS window in layer 0:

Layer-1 position t - W + 1 saw layer-0 positions [t - 2W + 2, t - W + 1].
Layer-1 position t - W + 2 saw layer-0 positions [t - 2W + 3, t - W + 2].
... and so on.

So when layer-2 attention at position t reads the layer-1 hidden state at position t - W + 1, it indirectly absorbs information from layer-0 positions as far back as t - 2W + 2. The effective reach at layer 2 is 2W positions.

L-layer reach

Iterating, after L layers the effective receptive field is approximately:

\text{RF}_{\text{theoretical}} \approx W \cdot L

for a depth-L sliding-window stack with window W.

Key insight: the per-layer window is the LOCAL constraint; depth is the LONG-RANGE mechanism. Without depth, sliding-window would be truncation; with depth, it is a multi-hop routing fabric.

Concrete numbers for Mistral 7B

The residual stream and why it makes depth-hops work

Theoretical vs effective receptive field, and the rise of hybrid architectures

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Attention pattern	Per-layer cost	Per-layer reach	Effective receptive field
Full attention	O(T^2 * d_k)	T (everything)	T, direct in one hop
Sliding window	O(W * T * d_k)	W tokens back	≈ W * L via depth-hops
SSM / Mamba	O(T * d_state)	Recurrent, all of history compressed	Bounded by d_state bandwidth
Hybrid (sliding + periodic full)	Mostly O(W * T) + sparse O(T^2)	W per layer, plus full-attention layers	≈ W * L with one-hop long edges at full-attn layers

Real products, models, and research that use this idea.

Mistral 7B: W = 4096, L = 32, theoretical receptive field ≈ 131k tokens despite nominal 8k pretraining context.
Longformer (Beltagy et al. 2020): pioneered sliding-window attention with W = 512 plus global tokens for document-level tasks.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the effective receptive field of a sliding-window model smaller than the theoretical W * L?

Each hop through depth has to repack the long-range signal into a fixed-dimension hidden state. That state already carries the local context, so distant content competes with nearby content for the same bandwidth. Empirically, sliding-window models trained on 4k contexts often degrade past 16-32k effective context length even when W * L is much larger. The information-bottleneck argument is what motivates hybrid architectures.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Assuming sliding-window models cannot see past W tokens. The per-layer window is W; the cross-layer receptive field is L * W via stacking.

60 second bullets to scan on the way to the call.

How layer stacking grows the receptive field beyond the per-layer window
The formula effective receptive field ≈ W * L

Primary sources. Browse if you want the original framing.