Match each sparse attention pattern to its design and exemplar architecture.

Match pairs·Hard·4.0 · 0·~2 min·Asked atCoinbase Lakera Qdrant

Attempt it

Drag each answer to line up with its matching prompt

Sliding window

Each token attends to W nearby tokens, local context only. Used in Mistral 7B (window=4096).

Dilated attention

Tokens grouped into blocks; intra block dense + cross-block sparse. Used in some GPT-3 ablations.

Global + local (BigBird style)

Skip-N pattern, attend to every Nth token over medium range. Used in Longformer.

Block sparse

Most tokens attend locally; a few designated 'global' tokens see and are seen by everyone, enabling long range info flow.

Standard attention fills the full n × n score matrix at `O(n²)`. **Sparse attention** computes only a subset. The designs differ in which subset. **Sliding window** keeps each query connected to W nearby tokens (512-4096 typical). The mask is a diagonal band. Distant tokens reach a query indirectly by hopping through layers. Mistral 7B uses W = 4096; Longformer uses it as its local component. **Dilated** extends the band with strided skips. Each query attends at `q-1, q-2, q-4, q-8`, reaching further than W with the same compute. Longformer uses dilated heads alongside local heads for medium range coverage. **Global plus local** (BigBird style) designates a small set as global. Each global attends to every other token and is attended by every other token, making globals a broadcast bus. Any two local tokens are at most two hops apart. BigBird adds random attention for its universal approximator argument. **Block sparse** groups the sequence into blocks of size B, dense within and sparse across via a strided or learned pattern. Sparse Transformer and GPT-3 ablations work this way; block structure aligns with GPU memory. By 2026, frontier LLMs default to exact dense attention via FlashAttention. Mistral's sliding window is the most prominent surviving sparse example.

Four families cover most of the sparse attention design space, each carving a different shape out of the n × n mask. **Sliding window** restricts each query to W nearby tokens, producing a diagonal band. Mistral 7B at W = 4096 is the canonical decoder example; Longformer uses it as its local component. The pattern is cheap and hardware-friendly but limited in per-layer reach. **Dilated** extends the band with strided skips (`q-1, q-2, q-4, q-8`) for medium range reach without paying for full global attention. Longformer combines dilated heads with local heads. **Global plus local**, the BigBird style, keeps local sliding window attention but designates a small set of global tokens that attend to every other token and are attended by every other token. Any two local tokens become at most two hops apart through a global. BigBird adds random attention on top for its universal approximator argument; Longformer uses local plus global without the random component. **Block sparse** does dense attention inside each block of size B and a sparse pattern across blocks. Sparse Transformer and some GPT-3 ablations live here; the block structure aligns naturally with GPU memory layouts and modern block sparse FlashAttention. Every family trades a piece of theoretical coverage for fewer computed pairs. By 2026 frontier LLMs default to exact FlashAttention plus ring attention plus YaRN.

TL;DR

Sparse attention patterns differ in which slice of the n × n grid they compute: diagonal band, strided band, band plus VIP tokens, or block structured.

Memory aid

Easy to grasp

Picture a huge grid asking, for each word, which other words it should look at. The standard kernel fills the whole grid. Sparse attention keeps most of the grid blank to save effort, and different designs pick different shapes to keep. A narrow band along the diagonal lets each word peek at nearby neighbors only, like reading with a small window. A strided band skips some neighbors but reaches further. A band plus a few full rows treats a handful of words as VIP guests everyone gets to talk to. Chunked patterns slice the sequence into groups and only check inside each group plus a few cross-links. Same idea everywhere, different geometry.

Key concepts

Sparse attention Long context

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Sparse attention is a family of structured masks on the n × n score matrix. The structure is chosen so the kernel computes only a subset of the entries, dropping the asymptotic cost while keeping enough connectivity for the model to remain expressive at sequence lengths dense attention cannot afford.

The four families split by the geometry of the mask. A diagonal band gives sliding window. A strided band gives dilated. A band plus a few full rows and columns gives the global plus local pattern. Chunks with sparse cross-chunk links give block sparse. The sections below walk each family along with its canonical exemplar architecture, distinguish sparse exact patterns from kernel approximations, then place the whole family inside the modern long context stack.

Sliding window: the diagonal band

Sliding window attention lets each query see W keys around it, producing a banded mask. Per-layer compute is O(n · W), linear in n, and the kernel is hardware-friendly because the band aligns with contiguous memory access.

Information propagates L · W tokens across L stacked layers; with Mistral 7B's W = 4096 and 32 layers, the theoretical reach is 131k positions. The catch is fidelity: information squeezes through intermediate tokens' residual streams at each hop, so distant relationships are weakly represented even when reach in principle is sufficient.

Mistral 7B with W = 4096 is the canonical modern decoder example, and the pattern continues in Mistral Large 3. Longformer uses sliding window as the local component of its heads, alongside dilated and global heads.

Dilated: the strided band

Global plus local: the broadcast bus

Block sparse: chunked geometry

Sparse exact vs kernel approximation

The modern long context stack

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Pattern	Geometry	Single layer reach	Exemplar
Sliding window	Diagonal band, width W	W tokens	Mistral 7B (W=4096)
Dilated	Strided band	W × stride	Longformer dilated heads
Global + local	Band + a few full rows/cols	Effectively n (via globals)	BigBird, Longformer
Block sparse	Dense blocks + sparse cross-block	n via cross-block pattern	Sparse Transformer, GPT-3 ablations

Real products, models, and research that use this idea.

Mistral Large 3 and the wider Mistral lineage continue to use sliding window attention (W around 4096) as a structural choice through 2026.
Longformer combines sliding window local plus dilated heads plus global tokens for documents up to 4096 tokens.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does BigBird need global tokens in addition to sliding window?

Pure sliding window attention can only propagate information across L × W tokens in L layers. For a 4096-token document with W=512 and L=12 layers, that's 6144, covered. For longer or more globally interactive tasks, you need a shortcut. Global tokens provide that: any local token reaches any other in 2 hops through a global one, regardless of distance.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Conflating 'sparse attention' with 'efficient attention', sparse attention is one family (approximate, fewer pairs computed); FlashAttention is exact (same pairs, different I/O). They're different layers of the stack.

60 second bullets to scan on the way to the call.

Geometry of each pattern: band, strided band, band with globals, blocks
Exemplar architectures: Mistral sliding window, Longformer, BigBird, Sparse Transformer

Primary sources. Browse if you want the original framing.