Sliding-window attention can't see beyond the window. Describe the trick that recovers older context.

The trick is **layer stacking**, and the carrier is the residual stream. At layer 1, position `t` directly attends to positions in `[t - W + 1, t]`, the literal window. So far so naive: no token older than `t - W + 1` is visible. At layer 2, position `t` again attends to its window, the layer-1 hidden states at positions `[t - W + 1, t]`. But each of those layer-1 hidden states is itself a function of ITS window, which extends one further window back. So through one hop, layer 2 at position `t` can be influenced by anything in `[t - 2W + 2, t]`. After `L` layers, the effective receptive field is roughly `W * L` positions. For Mistral-7B with `W = 4096` and `L = 32`, that is `4096 * 32 = 131,072` tokens, well past the nominal 8k context length. The model can be influenced by content from arbitrarily far back, just indirectly through depth-hops. The residual stream is what makes this work cleanly. Each layer's attention output is ADDED to the residual stream, not replacing it, so information that propagated forward at layer 5 is still available at layer 12 without needing a fresh attention edge. **The catch:** the long-range signal is indirect and bandwidth-limited. Each hop has to repack the relevant info into a single hidden state, and the model has to learn what is worth carrying forward.

**Layer stacking is the mechanism. Effective receptive field grows linearly with depth: roughly `W * L`.** At a single layer, sliding-window attention restricts each token to a window of `W` keys behind it. Layer 1 at position `t` can attend to `[t - W + 1, t]`, only the literal window. The key insight is what happens at layer 2. The hidden state at position `t - W + 1` in layer 1 is already a function of layer-0 positions in `[t - 2W + 2, t - W + 1]`, because that token's own layer-1 attention used a window of W keys behind it. So when layer-2 attention at position `t` reads the layer-1 hidden state at position `t - W + 1`, it indirectly absorbs information from positions up to `t - 2W + 2`. Iterating, after `L` layers the effective receptive field is approximately `W * L` positions. **Concrete numbers for Mistral 7B:** - Window `W = 4096` - Depth `L = 32` - Effective receptive field `≈ 4096 * 32 = 131,072 tokens` That is 16x larger than the model's nominal 8k pretraining context length. The model can absorb information from anywhere in a 130k-token input, despite no single attention layer ever looking past 4k. **The residual stream is the carrier.** Every layer's attention output is added to the residual stream rather than overwriting it, so once information has propagated forward at layer `k`, it remains available in the residual at all later layers without needing a fresh attention edge. **The trade-off versus full attention.** Sliding-window with depth-hops is far cheaper, `O(W * T * L)` instead of `O(T^2 * L)`, but the long-range signal is indirect and bandwidth-limited: each hop has to repack relevant info into a single hidden state. Full attention can route long-range information in one shot and remains more expressive when affordable.