Mistral 7B uses sliding window attention with a window of 4096 tokens. The model is fed a 16k-token input. What does the phrase 'effective context' mean for this configuration, and why is it larger than 4096?
Free · 2 AI evals / day
**Effective context means the longest input the model can actually integrate information across, not the per-layer attention window.**
Mistral 7B uses sliding-window attention with `W = 4096`. That means: at any single attention layer, each query position can attend only to the 4096 most recent key positions. Per layer, no token sees anything older than 4k positions back.
That sounds like it should cap the model at 4k context. It does not, because of layer stacking.
**The mechanism.** At layer 1, position `p` attends to positions in `[p - 4096 + 1, p]`. The output hidden state at position `p` is a function of those 4096 tokens.
At layer 2, position `p` again attends to its 4096-key window. But each of those layer-1 hidden states already absorbed information from positions up to 4096 further back. So layer 2 at position `p` can be influenced by content from up to `2 * 4096 = 8192` tokens behind.
After `L = 32` layers (Mistral 7B's depth), the theoretical receptive field is `32 * 4096 = 131,072` tokens, far more than the 16k input length. Information from anywhere in the 16k input can in principle reach the last token through depth-hops.
**The cost win.** Sliding-window's per-layer compute is `O(W * T) = O(4096 * 16k)`, much cheaper than full attention's `O(T^2) = O(16k^2)`. Mistral trades direct one-hop routing for cheap multi-hop routing through depth.
**Effective context for Mistral 7B over a 16k input is bounded by the cross-layer receptive field, roughly `W * L = 4096 * 32 = 131k tokens`, which exceeds the 16k input length, so every position can in principle influence every other position.**
The per-layer window `W = 4096` is the LOCAL constraint: at any single attention layer, each query attends only to the 4096 most recent keys. The CROSS-LAYER receptive field grows because information flows forward through each layer's window via the residual stream:
- Layer 1 at position p sees direct positions `[p - W + 1, p]`, reach W.
- Layer 2 at position p sees layer-1 hidden states across its window, each of which already absorbed W tokens behind it; reach `2W`.
- After L layers: reach approximately `L * W`.
For Mistral 7B (`W = 4096, L = 32`), the theoretical receptive field is `131k tokens`. Since `131k >> 16k`, every token in the 16k input can in principle propagate signal to the last token via depth-hops.
**Why this is called 'effective context' rather than 'attention window'.** The attention window is a per-layer property and is fixed at 4096. The effective context is the cross-layer reach, which is what actually matters for whether the model can integrate information from far-back tokens. The model can handle 16k inputs because the effective context bound exceeds 16k.
**The cost win versus full attention.** Per-layer compute is `O(W * T) = O(4096 * 16384)` instead of `O(T^2) = O(16384^2)`, roughly a 4x savings at 16k context. At longer contexts the savings compound: at 100k context, sliding-window costs `O(4k * 100k)`, full attention costs `O(100k^2)`, a 25x savings.
**The caveat:** the long-range signal is INDIRECT. Each hop has to repack the relevant information into the residual stream, so the practical effective context is bandwidth-limited. Empirically Mistral 7B degrades past 16-32k effective context on retrieval-heavy tasks even though the theoretical reach is 131k.
Quick reference
TL;DR
Per-layer reach is 4096; cross-layer receptive field is L * 4096. For Mistral 7B (32 layers) that is 131k tokens, larger than the 16k input, so every position can influence the last token.
Picture a row of librarians, each only able to read books from the four shelves in front of them. The librarian at the far right cannot directly read books from the far left of the room. But if the librarian at position 5 writes a one-page summary of what she sees, and the librarian at position 9 reads that summary as part of her own shelf range, the signal hops forward. Stack enough librarians in a line and the rightmost one can hear about books from the very left of the room, even though no single librarian ever saw more than four shelves at once.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Workloads requiring guaranteed direct attention from every position to every other position (theorem-proving, exact retrieval) Depth-hops are indirect and bandwidth-bottlenecked, so tasks that demand one-hop routing of arbitrary tokens need full attention or hybrid layers.
Inputs shorter than the sliding window (T < W) When the input is already smaller than 4096, sliding-window is a no-op and full attention is mathematically equivalent at the same cost.
Shallow models with L < 8 Cross-layer receptive field W * L is too small to span typical long inputs, defeating the depth-hop mechanism.
Comparing serving cost without accounting for KV cache Sliding-window's KV cache is O(W) per token regardless of sequence length, but the actual cost comparison must include cache management overhead and the cost of full-attention layers in hybrid stacks.
Quick scan · 2–4 min
Everything important, quickly.
Distinguish per-layer window W from cross-layer receptive field W * L, compute Mistral 7B's 131k bound, explain how the residual stream carries depth-hops, contrast cost with full attention, and flag the theoretical-vs-effective gap that motivates hybrid architectures.
Real-world examples
Real products, models, and research that use this idea.
1Mistral 7B and Mixtral: W = 4096 across 32 layers; theoretical receptive field ≈ 131k tokens over 8k pretraining context.
2Longformer (Beltagy et al. 2020): pioneered sliding-window with W = 512 plus global tokens for document-scale tasks.
3Gemini 3.1 Pro: hybrid stack interleaving sliding-window with periodic full-attention layers for million-token contexts.
4Jamba (AI21 2024): alternates Mamba SSM layers with sliding-window attention for long-context efficiency.
5Llama 3 and Llama 4 deliberately avoid sliding-window in favor of full attention with longer pretraining contexts, trading compute for one-hop routing.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does Mistral 7B's effective long-context performance degrade past ~20-30k tokens despite the theoretical 131k receptive field?
Sign in or go Pro to see the approach for this follow-up.
QHow does Mistral's sliding-window interact with its rolling KV cache, and what changes for serving compared to full attention?
Sign in or go Pro to see the approach for this follow-up.
QWhy do 2026 long-context stacks like Gemini 3.1 Pro and Jamba use hybrid attention patterns rather than pure sliding-window?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating 4096 as the model's maximum context. That is the per-layer window; the receptive field across 32 layers is 131k tokens.