Flash-Decoding parallelizes an axis FlashAttention v2 left alone. Pick which one.

Attempt it

Quick reference

TL;DR

Flash-Decoding splits the KV cache along the key axis so multiple SMs share the decode work, then reduces partials with log-sum-exp.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a librarian pulling one card from a giant filing cabinet to answer a question. With one librarian and a million cards, she does it alone and the line at the desk gets long. Flash-Decoding hires more librarians, splits the cabinet into drawers, and lets each librarian scan one drawer in parallel. They each pick their best candidate plus a confidence score, then a coordinator at the front combines the candidates into the final answer. The total amount of looking is the same, but the wall clock drops because the work spreads across people. That coordination, combining confidence scored partials into one answer, is what the log-sum-exp combine rule does.

Key concepts

FlashAttention Flash-Decoding KV cache Online softmax

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

60s: name the decode bottleneck (query axis vanishes), state the fix (split the key axis), explain the log-sum-exp combine is exact, give the sweet spot (long context + small batch), name production stacks (vLLM, TensorRT-LLM, SGLang).

Real-world examples

Real products, models, and research that use this idea.

vLLM ships a Flash-Decoding kernel path for long-context single-user serving on H100 and H200 hardware.
TensorRT-LLM integrates Flash-Decoding for Llama 4 Maverick and Qwen 3.5 deployments at 128k+ context.
SGLang uses Flash-Decoding for low-batch, long-context interactive serving where queue wait dominates.
DeepSeek V4 serving infrastructure documents Flash-Decoding alongside MLA for sub-second time to first token at million-token contexts.
The official tridao/flash-attention repo released Flash-Decoding as a separate kernel entry point in late 2023, now bundled in the FlashAttention v2.x line.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the log-sum-exp combine give an exact result rather than an approximation?

Sign in or go Pro to see the approach for this follow-up.

QWhen does Flash-Decoding stop helping, and what should you reach for instead?

Sign in or go Pro to see the approach for this follow-up.

QHow does Flash-Decoding interact with grouped-query attention?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming FlashAttention v2 already handles decode well. It optimizes prefill where many queries share K, V loads. Decode has one query, leaving SMs idle on long contexts.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.