Match each sparse attention pattern to its design and exemplar architecture.

Match pairs·Hard·4.0 · 0·~2 min

Attempt it

Sliding window

Dilated attention

Global + local (BigBird style)

Block sparse

Standard attention fills the full n × n score matrix at `O(n²)`. **Sparse attention** computes only a subset. The designs differ in which subset. **Sliding window** keeps each query connected to W nearby tokens (512-4096 typical). The mask is a diagonal band. Distant tokens reach a query indirectly by hopping through layers. Mistral 7B uses W = 4096; Longformer uses it as its local component. **Dilated** extends the band with strided skips. Each query attends at `q-1, q-2, q-4, q-8`, reaching further than W with the same compute. Longformer uses dilated heads alongside local heads for medium range coverage. **Global plus local** (BigBird style) designates a small set as global. Each global attends to every other token and is attended by every other token, making globals a broadcast bus. Any two local tokens are at most two hops apart. BigBird adds random attention for its universal approximator argument. **Block sparse** groups the sequence into blocks of size B, dense within and sparse across via a strided or learned pattern. Sparse Transformer and GPT-3 ablations work this way; block structure aligns with GPU memory. By 2026, frontier LLMs default to exact dense attention via FlashAttention. Mistral's sliding window is the most prominent surviving sparse example.

Four families cover most of the sparse attention design space, each carving a different shape out of the n × n mask. **Sliding window** restricts each query to W nearby tokens, producing a diagonal band. Mistral 7B at W = 4096 is the canonical decoder example; Longformer uses it as its local component. The pattern is cheap and hardware-friendly but limited in per-layer reach. **Dilated** extends the band with strided skips (`q-1, q-2, q-4, q-8`) for medium range reach without paying for full global attention. Longformer combines dilated heads with local heads. **Global plus local**, the BigBird style, keeps local sliding window attention but designates a small set of global tokens that attend to every other token and are attended by every other token. Any two local tokens become at most two hops apart through a global. BigBird adds random attention on top for its universal approximator argument; Longformer uses local plus global without the random component. **Block sparse** does dense attention inside each block of size B and a sparse pattern across blocks. Sparse Transformer and some GPT-3 ablations live here; the block structure aligns naturally with GPU memory layouts and modern block sparse FlashAttention. Every family trades a piece of theoretical coverage for fewer computed pairs. By 2026 frontier LLMs default to exact FlashAttention plus ring attention plus YaRN.

Quick reference

TL;DR

Sparse attention patterns differ in which slice of the n × n grid they compute: diagonal band, strided band, band plus VIP tokens, or block structured.

Memory aid

Easy to grasp

Picture a huge grid asking, for each word, which other words it should look at. The standard kernel fills the whole grid. Sparse attention keeps most of the grid blank to save effort, and different designs pick different shapes to keep. A narrow band along the diagonal lets each word peek at nearby neighbors only, like reading with a small window. A strided band skips some neighbors but reaches further. A band plus a few full rows treats a handful of words as VIP guests everyone gets to talk to. Chunked patterns slice the sequence into groups and only check inside each group plus a few cross-links. Same idea everywhere, different geometry.

Key concepts

Sparse attention Long context

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Walk each pattern with geometry, per-layer reach, and exemplar; explain BigBird's universal approximator motivation for global tokens; discuss why frontier LLMs use FlashAttention + ring + YaRN instead of heavy sparsity now.

Side-by-side

Pattern	Geometry	Single layer reach	Exemplar
Sliding window	Diagonal band, width W	W tokens	Mistral 7B (W=4096)
Dilated	Strided band	W × stride	Longformer dilated heads
Global + local	Band + a few full rows/cols	Effectively n (via globals)	BigBird, Longformer
Block sparse	Dense blocks + sparse cross-block	n via cross-block pattern	Sparse Transformer, GPT-3 ablations

Real-world examples

Real products, models, and research that use this idea.

Mistral Large 3 and the wider Mistral lineage continue to use sliding window attention (W around 4096) as a structural choice through 2026.
Longformer combines sliding window local plus dilated heads plus global tokens for documents up to 4096 tokens.
BigBird combines sliding window plus global plus random attention; designed for genomics and long documents.
OpenAI Sparse Transformer used factorized block sparse patterns; later GPT-class ablations adapted this.
Modern frontier models such as Llama 4 Maverick, Claude Opus 4.7, and Gemini 3.1 Pro generally favor exact attention plus FlashAttention plus position scaling over heavy sparsity for long context serving.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does BigBird need global tokens in addition to sliding window?

QWhy aren't modern frontier LLMs heavily sparse?

QHow does block sparse FlashAttention combine the two approaches?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Conflating 'sparse attention' with 'efficient attention', sparse attention is one family (approximate, fewer pairs computed); FlashAttention is exact (same pairs, different I/O). They're different layers of the stack.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.