You are designing a 40-layer decoder meant to serve 512k-token contexts on commodity GPUs. Pure global attention blows the KV budget; pure sliding window attention loses exact long range recall. Propose a concrete interleaving pattern of sliding window and global layers (with a ratio), justify it against what Gemma class models ship, and derive roughly how much KV memory your pattern saves versus all global.
Interleave 5 sliding-window layers per global layer, window 4096. At 512k context this gives a 5.6x KV cache shrink versus all-global, with global layers providing exact long-range recall.
Imagine a giant warehouse with 40 floors of file rooms. Most floors only need to see what is on a nearby shelf to answer their question, like 'what was just said.' Reserving every floor's view to the whole warehouse is wasteful because each floor would need a copy of every file. Better to give most floors a small local view (a few shelves) and only a few floors the big-picture full warehouse view. The full-warehouse floors handle the rare 'find the document from page 1' jobs. The local floors handle the bulk of work cheaply. This is what Gemma 3 does: most layers see only a window, a few see everything. The result is a building that works at the same quality but stores far less.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Long-context serving forces a hard trade-off. Pure global attention has perfect long-range recall but pays a KV cache cost linear in (sequence length) x (layer count). At 512k tokens and 40 layers, the bill is in the tens of gigabytes per sequence, which is the difference between fitting on a single GPU and not. Pure sliding-window attention has linear KV cost in (window size) x (layer count) but loses exact long-range recall: a token at position 500k cannot directly attend to a token at position 1 unless the relevant information has been relayed through many layers of local attention.
The shipping solution in 2026 is to interleave. Most layers use sliding-window attention because most token interactions are local. A few layers use global attention to handle the small fraction of interactions that actually need exact long-range recall. The Gemma 3 family chose 5:1 as their ratio after ablations; other models (Mistral, Phi-3) sit in a similar range.
This walkthrough covers the specific design (ratio, window size, placement), the KV cache math that justifies it, the receptive-field argument for why stacked local layers do useful work without global view, and the risk controls including attention sinks and eval discipline.
The 5:1 interleave pattern
Concretely for a 40-layer decoder: 33 sliding-window layers and 7 global layers, interleaved. One workable placement puts global layers at indices 5, 11, 17, 23, 29, 35, 39 (zero-indexed). The exact placement matters less than the spread: an early global layer ensures retrieval-dependent features have a chance to form, and a late global layer ensures the output decision incorporates the full context.
Local layers use a window of 4096 tokens. The choice is empirical: smaller windows lose paragraph-level structure, larger windows weaken the KV savings story. Gemma 3 ships in this neighborhood.
Global layers use full attention across the entire context. They are the only layers that can directly link arbitrary positions; everything else relays through stacked windows.
Why 5:1 specifically. Ablations in the Gemma 3 paper show needle-in-haystack accuracy stays close to all-global down to R=5:1, then drops sharply at R=8:1 and R=10:1. The ratio R=5:1 is the empirical Pareto frontier point: maximum local fraction (maximum KV savings) without measurable retrieval regression. Other model families (Mistral early sliding-window variants, Phi-3) sit in the R=4:1 to R=6:1 band, suggesting the band is robust across architectures.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Gemma 3 ships exactly this pattern: 5 sliding-window layers per global layer, window around 4096, for long-context efficiency.
- Mistral 7B v0.1 used sliding-window attention throughout, with effective receptive field growing through stacked layers; later variants added more global layers.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you adapt this pattern for a 1M-token target while keeping KV under 20 GB?
Either tighten the ratio further (8:1 or higher with attention-sink + landmark tokens) or combine with MLA/MQA on the global layers, or both. The trade is quality versus memory; needle-in-haystack at 1M is hard.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Picking too few global layers and tanking needle-in-haystack accuracy, or computing KV savings without accounting for the global layers still dominating at long context.
60 second bullets to scan on the way to the call.
The Gemma 3 5:1 local-to-global ratio and its window size
Why most interactions are local and a fraction need global view
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.