Match each sparse attention pattern to its design and exemplar architecture.

Attempt it

Sliding window

Dilated attention

Global + local (BigBird style)

Block sparse

Quick reference

TL;DR

Sparse attention patterns differ in which slice of the n × n grid they compute: diagonal band, strided band, band plus VIP tokens, or block structured.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a huge grid asking, for each word, which other words it should look at. The standard kernel fills the whole grid. Sparse attention keeps most of the grid blank to save effort, and different designs pick different shapes to keep. A narrow band along the diagonal lets each word peek at nearby neighbors only, like reading with a small window. A strided band skips some neighbors but reaches further. A band plus a few full rows treats a handful of words as VIP guests everyone gets to talk to. Chunked patterns slice the sequence into groups and only check inside each group plus a few cross-links. Same idea everywhere, different geometry.

Key concepts

Sparse attention Long context

Quick scan · 2–4 min

Everything important, quickly.

Walk each pattern with geometry, per-layer reach, and exemplar; explain BigBird's universal approximator motivation for global tokens; discuss why frontier LLMs use FlashAttention + ring + YaRN instead of heavy sparsity now.

Side-by-side

Pattern	Geometry	Single layer reach	Exemplar
Sliding window	Diagonal band, width W	W tokens	Mistral 7B (W=4096)
Dilated	Strided band	W × stride	Longformer dilated heads
Global + local	Band + a few full rows/cols	Effectively n (via globals)	BigBird, Longformer
Block sparse	Dense blocks + sparse cross-block	n via cross-block pattern	Sparse Transformer, GPT-3 ablations

Real-world examples

Real products, models, and research that use this idea.

Mistral Large 3 and the wider Mistral lineage continue to use sliding window attention (W around 4096) as a structural choice through 2026.
Longformer combines sliding window local plus dilated heads plus global tokens for documents up to 4096 tokens.
BigBird combines sliding window plus global plus random attention; designed for genomics and long documents.
OpenAI Sparse Transformer used factorized block sparse patterns; later GPT-class ablations adapted this.
Modern frontier models such as Llama 4 Maverick, Claude Opus 4.7, and Gemini 3.1 Pro generally favor exact attention plus FlashAttention plus position scaling over heavy sparsity for long context serving.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does BigBird need global tokens in addition to sliding window?

Sign in or go Pro to see the approach for this follow-up.

QWhy aren't modern frontier LLMs heavily sparse?

Sign in or go Pro to see the approach for this follow-up.

QHow does block sparse FlashAttention combine the two approaches?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.