Why is causal attention cheaper than full bidirectional at the same length?

Attempt it

Causal attention and full bidirectional attention compute on the same Q, K, V tensors with the same shapes. The savings come from a block-skipping trick in modern kernels. **The score matrix view.** Imagine the attention score matrix `S = Q K^T` as a square `(T, T)` grid where T is sequence length. For causal attention, the upper triangle is masked to `-inf` before softmax, those entries become exactly zero in the weight matrix and contribute nothing to the output. **What FlashAttention does.** FlashAttention processes the score matrix in tiles (typically 64-128 tokens per tile per dimension). For each tile, it checks where the tile sits relative to the causal diagonal: - Tile entirely below the diagonal: compute normally. - Tile straddling the diagonal: compute with a per-element mask applied. - Tile entirely above the diagonal: **skip entirely**. No matmul, no softmax, no memory traffic. Roughly half of the tiles fall entirely above the diagonal, so causal attention does about half the work of bidirectional attention at the same length. **Why option A is wrong:** Q, K, V have the same `(T, d_model)` shapes in both cases. **Why option C is wrong:** cross-head sharing (MQA/GQA) is orthogonal, it shrinks the cache, not the matmul. **Why option D is wrong:** both causal and bidirectional use softmax.

Causal attention is cheaper than bidirectional in modern FlashAttention-style kernels because **the kernel skips computing tiles that fall entirely above the causal diagonal**. FlashAttention processes the attention score matrix in tiles (typically 64-128 tokens per dimension). For a sequence of length T, the score matrix has roughly `(T / 64)^2` tiles. The kernel iterates over these tiles to compute the fused matmul + softmax + V-matmul block by block, never materializing the full score matrix in high-bandwidth memory. ### Three tile categories 1. **Tiles entirely below the diagonal**, every entry is a legal causal attention. Compute normally. 2. **Tiles straddling the diagonal**, some entries are masked, some are not. Compute with an in-tile mask. 3. **Tiles entirely above the diagonal**, every entry would be `-inf` after the causal mask, contributing zero to softmax. The kernel skips them entirely: no matmul, no softmax, no memory traffic. Roughly half the tiles fall in category 3, giving causal attention about a **2x speedup** over bidirectional at the same sequence length. FlashAttention-2 with `is_causal=True` and FlashAttention-3 on H100 both implement this skip above diagonal optimization. The other options confuse different concepts: - **Q, K, V shapes are identical** in causal and bidirectional attention, option A is false. - **Cross-head K, V sharing** is MQA/GQA, orthogonal to causal masking, option C is false. - **Softmax is used in both**, option D is false.

Quick reference

TL;DR

FlashAttention skips entire blocks above the causal diagonal because they would softmax to zero, roughly half the work compared to bidirectional, where every block must be computed.

Memory aid

Easy to grasp

Picture filling out a square grid of homework problems. Bidirectional attention is doing every single problem in the grid. Causal attention has a rule: any problem above the diagonal is just zero, do not write anything. A clever student skips those squares entirely instead of writing zero in each one. FlashAttention is that clever student, it inspects the grid tile by tile, sees which tiles are entirely above the diagonal, and skips them without doing the math. Since roughly half the grid is above the diagonal, the clever student does half the work for the same result.

Key concepts

FlashAttention Causal mask GPU kernel optimization

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

FlashAttention processes the score matrix in tiles and skips tiles entirely above the causal diagonal. Roughly half the tiles fall above, giving ~2x kernel speedup over bidirectional at the same length. Q, K, V shapes are identical; the savings are in tile scheduling, not matrix shape.

Side-by-side

Aspect	Bidirectional	Causal (with FlashAttention)
Q, K, V shapes	(T, d_model)	(T, d_model)
Score matrix tiles computed	All (T/ts)^2	Roughly half (lower triangle + diagonal)
Softmax	Row-wise on full row	Row-wise on causal prefix
Relative kernel cost	1x baseline	~0.5x
Skip mechanism	None	FlashAttention tile scheduler

Real-world examples

Real products, models, and research that use this idea.

FlashAttention-2 (Dao 2023): introduced causal-aware tile scheduling; the `is_causal=True` flag activates the upper-triangle skip.
FlashAttention-3 (Shah et al. 2024): brings the same optimization plus warp-specialization to H100 GPUs.
Llama 4 Maverick and Mistral Large 2 serving via vLLM with FlashAttention-3: benchmark roughly 2x faster per-token throughput on the attention kernel versus bidirectional at the same length.
PyTorch's `F.scaled_dot_product_attention(..., is_causal=True)`: dispatches to the causal-aware FlashAttention path when available.
PagedAttention in vLLM: KV pages for future positions are never allocated, so the kernel does not even iterate over them, a related but distinct optimization.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf causal attention is 2x faster on the attention kernel, why is end to end transformer block throughput only 1.3-1.5x faster instead of 2x?

QHow does sliding-window causal attention (Mistral 7B) change the FlashAttention tile skip pattern?

QWhy can't bidirectional attention exploit any block-skipping optimization in FlashAttention?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Believing causal attention is cheaper because the matrices are smaller. The Q, K, V shapes are identical; the savings come from skipping blocked-out blocks of the score matrix.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.