Match each sparse attention pattern to its design and exemplar architecture.

Match pairs·Hard·4.0 · 0·~2 min·Asked atCoinbase Lakera Qdrant

Attempt it

Drag each answer to line up with its matching prompt

Sliding window

Each token attends to W nearby tokens, local context only. Used in Mistral 7B v0.1 (window=4096).

Dilated attention

Tokens grouped into blocks; intra block dense + cross-block sparse. Used in Sparse Transformer; GPT-3 alternates such sparse layers with dense ones.

Global + local (BigBird style)

Skip-N pattern, attend to every Nth token over medium range. Used in Longformer.

Block sparse

Most tokens attend locally; a few designated 'global' tokens see and are seen by everyone, enabling long range info flow.

Standard attention fills the full n × n score matrix at `O(n²)`. **Sparse attention** computes only a subset. The designs differ in which subset. **Sliding window** keeps each query connected to W nearby tokens (512-4096 typical). The mask is a diagonal band. Distant tokens reach a query indirectly by hopping through layers. Mistral 7B v0.1 uses W = 4096; Longformer uses it as its local component. **Dilated** extends the band with strided skips. Each query attends to every d-th token (`q-4, q-8, q-12` for d = 4), reaching further than W with the same compute. Longformer mixes dilated and local heads. **Global plus local** (BigBird style) designates a small set as global. Each global attends to, and is attended by, every other token, like a broadcast bus. Any two local tokens are at most two hops apart. BigBird also adds a few random links. **Block sparse** groups the sequence into blocks of size B, dense within and sparse across via a strided or learned pattern. Sparse Transformer works this way; block structure aligns with GPU memory. Frontier LLMs now default to exact dense attention via FlashAttention. Some models, such as Gemma 2, still mix in sliding window layers.

Four families cover most of the sparse attention design space, each carving a different shape out of the n × n mask. **Sliding window** restricts each query to W nearby tokens, producing a diagonal band. Mistral 7B v0.1 at W = 4096 is the standard decoder example; Longformer uses it as its local component. The pattern is cheap and hardware-friendly but limited in reach per layer. **Dilated** extends the band with strided skips (every d-th token, such as `q-4, q-8, q-12`) for medium range reach without paying for full global attention. Longformer combines dilated heads with local heads. **Global plus local**, the BigBird style, keeps local sliding window attention but designates a small set of global tokens. These tokens attend to every other token and are attended by every other token. Any two local tokens become at most two hops apart through a global. BigBird adds random attention on top, though its universal approximator proof needs only the global tokens. Longformer uses local plus global without the random component. **Block sparse** does dense attention inside each block of size B and a sparse pattern across blocks. Sparse Transformer lives here, and GPT-3 alternates such sparse layers with dense ones. The block structure aligns naturally with GPU memory layouts and modern block sparse FlashAttention. Every family trades a piece of theoretical coverage for fewer computed pairs. Frontier LLMs now default to exact FlashAttention plus ring attention plus YaRN.

TL;DR

Sparse attention patterns differ in which slice of the n × n grid they compute: diagonal band, strided band, band plus VIP tokens, or block structured.

Memory aid

Easy to grasp

Picture a huge grid asking, for each word, which other words it should look at. The standard kernel fills the whole grid. Sparse attention keeps most of the grid blank to save effort, and different designs pick different shapes to keep. A narrow band along the diagonal lets each word peek at nearby neighbors only, like reading with a small window. A strided band skips some neighbors but reaches further. A band plus a few full rows treats a handful of words as VIP guests everyone gets to talk to. Chunked patterns slice the sequence into groups and only check inside each group plus a few cross-links. The idea is the same everywhere; only the geometry differs.

Key concepts

Sparse attention Long context

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Sparse attention is a family of structured masks on the n × n score matrix. The structure is chosen so the kernel computes only a subset of the entries. This drops the asymptotic cost while keeping enough connectivity for the model to remain expressive at sequence lengths dense attention cannot afford.

The four families split by the geometry of the mask. A diagonal band gives sliding window. A strided band gives dilated. A band plus a few full rows and columns gives the global plus local pattern. Chunks with sparse cross-chunk links give block sparse.

Sliding window: the diagonal band

Sliding window attention lets each query see W keys around it, producing a banded mask. Compute per layer is O(n · W), linear in n, and the kernel is hardware-friendly because the band aligns with contiguous memory access.

Information propagates L · W tokens across L stacked layers. With Mistral 7B's W = 4096 and 32 layers, the theoretical reach is 131k positions. The catch is fidelity: information squeezes through intermediate tokens' residual streams at each hop. So distant relationships are weakly represented even when reach in principle is sufficient.

Mistral 7B v0.1 with W = 4096 is the standard decoder example. Later Mistral models dropped it for dense attention, while Gemma 2 alternates sliding window layers with global layers. Longformer uses sliding window as the local component of its heads, alongside dilated and global heads.

Dilated: the strided band

Global plus local: the broadcast bus

Block sparse: chunked geometry

Sparse exact vs kernel approximation

The modern long-context stack

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Pattern	Geometry	Single layer reach	Exemplar
Sliding window	Diagonal band, width W	W tokens	Mistral 7B v0.1 (W=4096)
Dilated	Strided band	W × stride	Longformer dilated heads
Global + local	Band + a few full rows/cols	Effectively n (via globals)	BigBird, Longformer
Block sparse	Dense blocks + sparse cross-block	n via cross-block pattern	Sparse Transformer, GPT-3 (alternating with dense layers)

Real products, models, and research that use this idea.

Mistral 7B v0.1 used sliding window attention with W = 4096. Later Mistral models such as Mixtral dropped it for dense attention, while Gemma 2 alternates 4096-token sliding window layers with global layers.
Longformer combines sliding window local plus dilated heads plus global tokens for documents up to 4096 tokens.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does BigBird need global tokens in addition to sliding window?

In a bidirectional encoder, a window of W covers W/2 tokens per side, so pure sliding window attention propagates information only about L × W/2 tokens in each direction across L layers. For a 4096-token document with W=512 and L=12 layers, that is 3072 tokens, so the first and last tokens never connect. For longer or more globally interactive tasks, you need a shortcut. Global tokens provide that: any local token reaches any other in 2 hops through a global one, regardless of distance.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Conflating 'sparse attention' with 'efficient attention'. Sparse attention is one family (approximate, fewer pairs computed); FlashAttention is exact (same pairs, different I/O). They're different layers of the stack.

60 second bullets to scan on the way to the call.

Geometry of each pattern: band, strided band, band with globals, blocks
Exemplar architectures: Mistral sliding window, Longformer, BigBird, Sparse Transformer

Primary sources. Browse if you want the original framing.