What's the structural limitation of pure sliding window attention, and how does BigBird address it?
Sliding window attention is local-only; long range info crawls layer by layer. BigBird adds global tokens so any pair is reachable in two hops.
Picture a long line of people passing notes, where each person can only talk to their immediate neighbors. A note from one end takes many small steps to reach the other end, and it can get garbled along the way. BigBird gives a few people in the line megaphones. Anyone in the line can shout to a megaphone holder, and that megaphone holder can shout back to anyone. Now a message from any position can reach any other in one quick step through a megaphone. You still keep the local chatter, which is cheap, and you also get the broadcast capability for the important stuff.
Detailed answer & concept explanation~6 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Explain sliding window's O(n·W) complexity and the multi-hop propagation limitation, walk through BigBird's three components (local + global + random) with the expressivity theorem, distinguish from Longformer, mention Mistral 7B's pure sliding window design, and discuss why dense attention + FlashAttention has largely replaced sparse designs for modern long context LLMs.
| Pattern | Complexity | Long range flow | Examples |
|---|---|---|---|
| Full attention | O(n²) | Direct any to any | Standard transformer |
| Sliding window only | O(n·W) | Multi-hop through layers (lossy) | Mistral 7B |
| Local + global (Longformer) | O(n·W + n·G) | 2-hop via global tokens | Longformer |
| Local + global + random (BigBird) | O(n·(W+G+R)) | 2-hop via global, more connectivity | BigBird |
| Dilated (LongNet) | O(n log n) | Multi-scale dilation | LongNet |
Real products, models, and research that use this idea.
- BigBird: used for long document NLP tasks like NaturalQuestions and ArXiv summarization at 4096-8192 tokens.
- Longformer: widely adopted for long document tasks; 'local plus global' template without random attention.
- Mistral Large 3 continues to use sliding window attention as a core architectural choice; relies on layer depth for long range propagation.
- Sparse Transformer: uses strided plus local patterns; one of the first sub-quadratic attention designs.
- Frontier 2026 long context stacks for Llama 4 Maverick, Claude Opus 4.7, and Gemini 3.1 Pro generally prefer dense attention plus FlashAttention plus RoPE scaling over heavy sparsity.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy has dense attention with FlashAttention largely replaced sparse attention for long context LLMs?
QHow does Mistral 7B's pure sliding window design handle long range dependencies?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating sliding window as a drop in efficient replacement for full attention. It is, until you need long range dependencies, and then the multi-hop information path through intermediate tokens becomes the new bottleneck.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.