Flash-Decoding splits the KV cache along the key axis so multiple SMs share the decode work, then reduces partials with log-sum-exp.
Picture a librarian pulling one card from a giant filing cabinet to answer a question. With one librarian and a million cards, she does it alone and the line at the desk gets long. Flash-Decoding hires more librarians, splits the cabinet into drawers, and lets each librarian scan one drawer in parallel. They each pick their best candidate plus a confidence score, then a coordinator at the front combines the candidates into the final answer. The total amount of looking is the same, but the wall clock drops because the work spreads across people. That coordination, combining confidence scored partials into one answer, is what the log-sum-exp combine rule does.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
FlashAttention v2 rewrote the I/O pattern for attention so the n-by-n matrix never lives in HBM. That single change made long-context training and prefill tractable on a single H100. But decode, the autoregressive token by token generation that drives every chatbot, broke the assumptions that made v2 fast.
Flash-Decoding is the targeted fix. It does not replace FlashAttention v2; it adds a third parallelism axis exactly where v2 ran out of work, the key sequence axis. The sections below walk why decode is structurally different from prefill, how splitting the K, V cache restores SM occupancy, why the cross-chunk combine is exact rather than an approximation, where the win shows up in production serving, and where it stops helping.
Why decode breaks FlashAttention v2
FlashAttention v2 parallelizes over the query sequence axis and the head axis. During prefill, with a 4k-token prompt and 32 attention heads, you have 4096 * 32 independent work items, easily enough to saturate an H100's 132 streaming multiprocessors several times over.
What decode looks like
Decode emits one token per forward pass. The attention call has Q of shape (batch, n_heads, 1, d_head) against K and V of shape (batch, n_kv_heads, T, d_head) where T is the current KV cache length. The query sequence axis is now length 1. It is not a parallelism source; it is a single point.
With batch 1 and 32 heads, you have 32 work items for 132 SMs. Roughly 100 SMs sit idle. The kernel finishes the per-SM work quickly, then stalls waiting for HBM bandwidth to stream in the long KV cache, which can be hundreds of megabytes at 128k context.
The decode bottleneck is not compute. It is SM occupancy combined with memory bandwidth, and FlashAttention v2's two parallel axes do not give the scheduler enough work.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM ships a Flash-Decoding kernel path for long-context single-user serving on H100 and H200 hardware.
- TensorRT-LLM integrates Flash-Decoding for Llama 4 Maverick and Qwen 3.5 deployments at 128k+ context.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the log-sum-exp combine give an exact result rather than an approximation?
Softmax(x) is shift-invariant: subtracting the same constant from every x_i leaves the result unchanged. The trick partitions the keys into chunks, computes per-chunk softmax denominators with chunk-local maxes, then rescales each chunk's contribution by exp(m_chunk - m_global) before summing. The rescaling exactly aligns the chunks to the global max, so the combined denominator is the true global denominator and the combined numerator is the true global weighted sum.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming FlashAttention v2 already handles decode well. It optimizes prefill where many queries share K, V loads. Decode has one query, leaving SMs idle on long contexts.
60 second bullets to scan on the way to the call.
Why decode kills FlashAttention v2's query-axis parallelism
The three axes available during attention and which one Flash-Decoding adds
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.