FlashAttention

Pair HBM and SRAM with their roles on an inference GPU

How does torch.nn.functional.scaled_dot_product_attention relate to FlashAttention?

Does FlashAttention change the attention output relative to standard attention?

Flash-Decoding parallelizes an axis FlashAttention v2 left alone. Pick which one.

In a 7B transformer, do attention or MLP layers eat more FLOPs, and does context length change the answer?

Why does FlashAttention speed up serving even though it doesn't change the math?

Where does FlashAttention deliver the biggest serving speedup?

Why is standard attention O(n²) in sequence length, and what specifically is the n²?

Where in attention is FP32 still used, and what breaks if you push everything to FP16/BF16?

PremiumWhy is standard attention…

Walk through what FlashAttention does differently from standard attention, and what changes between v1, v2, and v3.

What does FlashAttention v1 actually change vs standard attention?

Match each long context strategy to what it modifies.