Mistral 7B uses sliding-window attention with a 4k window over 32k context; Llama-2 7B uses full causal attention over 4k. Same parameter count, different attention pattern.
Imagine two researchers asked to find connections in a thousand-page book. The first researcher reads page N by glancing at every page before it, a slow read but every page sees every prior page. The second researcher reads page N by glancing only at the last 100 pages, but does this many times in stacked passes; by the final pass, information from page 1 has hopped through enough intermediate pages to reach page N. The first researcher is Llama-2 7B with full attention; the second is Mistral 7B with sliding-window attention.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Mistral 7B and Llama-2 7B landed within months of each other in late 2023 and looked, on a parameter sheet, almost identical: same scale, same RoPE, both decoder-only causal transformers. Yet their per-token serving cost and context handling differ sharply, and the reason is one architectural choice that is easy to miss in a cursory comparison.
Llama-2 7B uses full causal attention with a 4k context. Mistral 7B uses sliding-window attention with a 4k window inside a 32k total context. That single pattern difference cascades into per-token cost, KV cache behavior, effective receptive field, and the kinds of long-context tasks each model handles well.
This deep dive walks the pattern difference, the cost scaling, the depth-stacking trick that grows Mistral's effective context past its window, and the long-context evaluation regime where the asymmetry shows up.
Two patterns, one parameter count
On the surface, the two models look nearly identical. Same 7B scale. Both decoder-only causal transformers. Both use RoPE for positional encoding. Both use SwiGLU and pre-norm.
Where they differ
The attention pattern. Llama-2 7B is full causal: at every layer, every token attends to every prior token in the 4k context. Mistral 7B is sliding-window: at every layer, every token attends to a window of the 4k most recent tokens within a 32k total context.
A secondary difference
Mistral 7B uses GQA (32 query heads, 8 KV heads). Llama-2 7B uses standard MHA (32 query heads, 32 KV heads). This shrinks Mistral's KV cache 4x relative to Llama-2 7B's per-token cache, but it is not the headline architectural difference. GQA appears in Llama-2 at 34B and 70B; it is independent of the sliding-window choice.
The sliding-window choice is what enables Mistral 7B's 32k context at 7B scale. GQA helps; window attention is the architectural foundation.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | Llama-2 7B | Mistral 7B |
|---|---|---|
| Attention pattern | Full causal | Sliding window (4096) |
| Context length | 4096 | 32768 |
| KV head structure | MHA (32 query, 32 KV) | GQA (32 query, 8 KV) |
| Positional encoding | RoPE | RoPE |
| Per-token attention cost | O(seq) | O(window) |
| Per-layer receptive field | Full prior context | 4096 nearest tokens |
| Effective receptive field | Same as per-layer | L * window via stacking |
Real products, models, and research that use this idea.
- Mistral 7B's release blog explicitly documents the 4096-token sliding window inside a 32768 total context.
- Llama-2 7B's release paper specifies 4k full causal attention and standard MHA.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does sliding-window attention interact with the KV cache at long context?
Tokens outside the window for the current step can be evicted from cache, bounding cache size by window * H_kv * d_h * L. This is what makes the 32k context affordable at 7B.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Naming MQA, GQA, or RoPE as the headline difference. Mistral 7B and Llama-2 7B differ in attention PATTERN (full vs sliding window), not in head sharing or positional encoding.
60 second bullets to scan on the way to the call.
The headline difference: full vs sliding-window attention
Mistral 7B window size (4096) and total context (32768)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.