Both share the same motivation: standard attention is `O(n²)`, prohibitive at 8k, 16k, or 100k tokens.
**Sliding window** lets each token see W neighbors on each side. Only entries within W of the diagonal are nonzero. Compute drops to `O(n · W)`, linear in n. For 16k tokens at W = 512 that is ~16M entries instead of 256M, a 16x saving.
The cost shows up on long range relationships. With W = 512, information from token 0 reaches token 512 after one layer and `L · W` positions after L layers. With 24 layers that is 12k positions, barely enough for 16k context, and every hop loses fidelity. Many semantic relationships need direct connections, not telephone chains.
**BigBird** addresses this with three combined patterns. Sliding window to `±W` neighbors captures local structure. Global tokens (a small set) attend to and are attended by every position, so any two positions are at most two hops apart. Random attention adds a few extra links for connectivity. BigBird's paper proves this combination is a universal approximator.
The overall pattern stays sparse: `O(n · (W + G + R))`.
**Pure sliding window attention** restricts each token to `±W` local neighbors. The kernel is cheap, but long range information has to crawl through multiple layer hops to reach distant tokens, and every hop loses fidelity through the intermediate token's residual stream.
BigBird fixes this by combining three patterns. Local sliding window attention carries short range structure at `O(n · W)`. Global tokens, a small designated set G, attend to and are attended by every position, so any two tokens become two hops apart through a global at `O(n · G)` extra cost. Random attention adds a few extra links per query for theoretical connectivity at `O(n · R)`.
The global tokens do the heavy lifting on long range reach; the local and random pieces fill in efficiency and expressivity. BigBird's paper proves the combined pattern is a universal approximator over sequence functions, a property pure sliding window lacks. Longformer uses local plus global without the random component and ended up more widely adopted. Mistral 7B takes the other path: sliding window only, no globals, relying on depth.
Quick reference
TL;DR
Sliding window attention is local-only; long range info crawls layer by layer. BigBird adds global tokens so any pair is reachable in two hops.
Picture a long line of people passing notes, where each person can only talk to their immediate neighbors. A note from one end takes many small steps to reach the other end, and it can get garbled along the way. BigBird gives a few people in the line megaphones. Anyone in the line can shout to a megaphone holder, and that megaphone holder can shout back to anyone. Now a message from any position can reach any other in one quick step through a megaphone. You still keep the local chatter, which is cheap, and you also get the broadcast capability for the important stuff.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tasks needing direct token to token interactions across the full sequence in a single layer Sliding window's reach is bounded by W per layer; multi-hop propagation through intermediates is lossy and slow to converge on hard long range dependencies.
Frontier general purpose LLMs in 2026 Production stacks lean on exact attention plus FlashAttention plus RoPE scaling; sparse patterns trade expressivity for cost savings that matter less when memory is no longer the binding constraint.
Short sequences where dense attention already fits At small n, the constant factor overhead of sparse kernels and the expressivity loss are not worth the negligible compute savings.
Workloads that need exact attention weights for interpretability Sparse kernels never materialize the dropped pairs; downstream analysis tools that scan the full n by n matrix will miss real signal.
Domains with non-stationary or irregular long range dependency structure Fixed sparsity patterns (band, dilated, global plus local) bake in assumptions; tasks that break those assumptions regress relative to dense attention.
Quick scan · 2–4 min
Everything important, quickly.
Explain sliding window's O(n·W) complexity and the multi-hop propagation limitation, walk through BigBird's three components (local + global + random) with the expressivity theorem, distinguish from Longformer, mention Mistral 7B's pure sliding window design, and discuss why dense attention + FlashAttention has largely replaced sparse designs for modern long context LLMs.
Side-by-side
Pattern
Complexity
Long range flow
Examples
Full attention
O(n²)
Direct any to any
Standard transformer
Sliding window only
O(n·W)
Multi-hop through layers (lossy)
Mistral 7B
Local + global (Longformer)
O(n·W + n·G)
2-hop via global tokens
Longformer
Local + global + random (BigBird)
O(n·(W+G+R))
2-hop via global, more connectivity
BigBird
Dilated (LongNet)
O(n log n)
Multi-scale dilation
LongNet
Real-world examples
Real products, models, and research that use this idea.
1BigBird: used for long document NLP tasks like NaturalQuestions and ArXiv summarization at 4096-8192 tokens.
2Longformer: widely adopted for long document tasks; 'local plus global' template without random attention.
3Mistral Large 3 continues to use sliding window attention as a core architectural choice; relies on layer depth for long range propagation.
4Sparse Transformer: uses strided plus local patterns; one of the first sub-quadratic attention designs.
5Frontier 2026 long context stacks for Llama 4 Maverick, Claude Opus 4.7, and Gemini 3.1 Pro generally prefer dense attention plus FlashAttention plus RoPE scaling over heavy sparsity.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy has dense attention with FlashAttention largely replaced sparse attention for long context LLMs?
Sign in or go Pro to see the approach for this follow-up.
QHow does Mistral 7B's pure sliding window design handle long range dependencies?
Sign in or go Pro to see the approach for this follow-up.
QWhat's the theoretical justification for random attention in BigBird?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating sliding window as a drop in efficient replacement for full attention. It is, until you need long range dependencies, and then the multi-hop information path through intermediate tokens becomes the new bottleneck.