Identify the attention-pattern difference between Mistral 7B and Llama-2 7B

Mistral 7B and Llama-2 7B are both 7-billion-parameter open-source language models that came out within a few months of each other and look very similar on the surface. The headline difference between them is the shape of their attention. Llama-2 7B uses **full causal attention**. Every token at every layer attends to every previous token in the context. The context length is 4096 tokens. Mistral 7B uses **sliding-window attention**. Every token at every layer attends to a fixed window of the most recent 4096 tokens, even though the total context length is 32768. So a token at position 20000 sees positions 15904 to 20000, not all 20000 prior tokens. Mistral makes up for the limited per-layer reach by stacking many layers: information from far back can hop through several layers' worth of 4k windows to reach a token in a later layer. The trade is cheaper per-token attention and a much longer total context, at the cost of slightly less direct long-range reach in any single layer. Both models use RoPE for positional encoding and both use grouped variants (Llama-2 7B is MHA, Mistral 7B is GQA), but those are not the headline pattern difference.

The defining attention-pattern difference is full versus sliding window. Llama-2 7B uses full causal attention over 4096 tokens; Mistral 7B uses sliding-window attention with a window of 4096 tokens inside a 32768-token total context. With full attention, per-token cost scales linearly with sequence length on reads (and the KV cache grows linearly), so doubling context doubles work and memory. With sliding-window attention, per-token cost is bounded by the window size, which makes a 32k context affordable at 7B parameters even on consumer hardware. The trick that keeps Mistral's effective receptive field large despite the 4k window is depth-stacking: information from a far-back position can hop window by window through stacked layers. After L layers, the effective receptive field is roughly `L * window`, which on a 32-layer model is well past 32k. Both models use RoPE for positional encoding, so positional choice is not the headline difference. Llama-2 7B uses standard MHA; Mistral 7B uses GQA with 8 KV heads. The MHA vs GQA difference matters for KV cache size, but the attention pattern difference (full vs sliding window) is what enables Mistral's 32k context at 7B scale in the first place.