Identify the attention-pattern difference between Mistral 7B and Llama-2 7B

Attempt it

Quick reference

TL;DR

Mistral 7B uses sliding-window attention with a 4k window over 32k context; Llama-2 7B uses full causal attention over 4k. Same parameter count, different attention pattern.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine two researchers asked to find connections in a thousand-page book. The first researcher reads page N by glancing at every page before it, a slow read but every page sees every prior page. The second researcher reads page N by glancing only at the last 100 pages, but does this many times in stacked passes; by the final pass, information from page 1 has hopped through enough intermediate pages to reach page N. The first researcher is Llama-2 7B with full attention; the second is Mistral 7B with sliding-window attention.

Key concepts

Sliding-window attention KV cache

Quick scan · 2–4 min

Everything important, quickly.

5m: the full vs sliding-window pattern split, Mistral's 4k window inside 32k context, depth-stacking to grow effective receptive field, what GQA and RoPE do and don't change, and where sliding-window costs you on long-range lookup.

Side-by-side

Property	Llama-2 7B	Mistral 7B
Attention pattern	Full causal	Sliding window (4096)
Context length	4096	32768
KV head structure	MHA (32 query, 32 KV)	GQA (32 query, 8 KV)
Positional encoding	RoPE	RoPE
Per-token attention cost	O(seq)	O(window)
Per-layer receptive field	Full prior context	4096 nearest tokens
Effective receptive field	Same as per-layer	L * window via stacking

Real-world examples

Real products, models, and research that use this idea.

Mistral 7B's release blog explicitly documents the 4096-token sliding window inside a 32768 total context.
Llama-2 7B's release paper specifies 4k full causal attention and standard MHA.
FlashAttention 2 added efficient sliding-window kernels that Mistral 7B inference relies on.
vLLM and llama.cpp both support sliding-window attention via attention mask flags when serving Mistral models.
Later Mistral models (Mistral Large 3) moved to a hybrid of sliding-window and full attention layers to get the best of both.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does sliding-window attention interact with the KV cache at long context?

Sign in or go Pro to see the approach for this follow-up.

QWhy did Mistral Large 3 move to a hybrid full and sliding-window layer pattern?

Sign in or go Pro to see the approach for this follow-up.

QWhat evaluation would expose Mistral's pattern weakness vs Llama-2 at long context?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.