Why is causal attention cheaper than full bidirectional at the same length?

MCQ·Medium·4.0 · 0·~1 min·Asked atNetflix Typeface Wandb·Relevant atAi4bharat Cerebras NVIDIA Replicate

Attempt it

Causal attention and full bidirectional attention compute on the same Q, K, V tensors with the same shapes. The savings come from a block-skipping trick in modern kernels. **The score matrix view.** Imagine the attention score matrix `S = Q K^T` as a square `(T, T)` grid where T is sequence length. For causal attention, the upper triangle is masked to `-inf` before softmax, those entries become exactly zero in the weight matrix and contribute nothing to the output. **What FlashAttention does.** FlashAttention processes the score matrix in tiles (typically 64-128 tokens per tile per dimension). For each tile, it checks where the tile sits relative to the causal diagonal: - Tile entirely below the diagonal: compute normally. - Tile straddling the diagonal: compute with a per-element mask applied. - Tile entirely above the diagonal: **skip entirely**. No matmul, no softmax, no memory traffic. Roughly half of the tiles fall entirely above the diagonal, so causal attention does about half the work of bidirectional attention at the same length. **Why option A is wrong:** Q, K, V have the same `(T, d_model)` shapes in both cases. **Why option C is wrong:** cross-head sharing (MQA/GQA) is orthogonal, it shrinks the cache, not the matmul. **Why option D is wrong:** both causal and bidirectional use softmax.

Causal attention is cheaper than bidirectional in modern FlashAttention-style kernels because **the kernel skips computing tiles that fall entirely above the causal diagonal**. FlashAttention processes the attention score matrix in tiles (typically 64-128 tokens per dimension). For a sequence of length T, the score matrix has roughly `(T / 64)^2` tiles. The kernel iterates over these tiles to compute the fused matmul + softmax + V-matmul block by block, never materializing the full score matrix in high-bandwidth memory. ### Three tile categories 1. **Tiles entirely below the diagonal**, every entry is a legal causal attention. Compute normally. 2. **Tiles straddling the diagonal**, some entries are masked, some are not. Compute with an in-tile mask. 3. **Tiles entirely above the diagonal**, every entry would be `-inf` after the causal mask, contributing zero to softmax. The kernel skips them entirely: no matmul, no softmax, no memory traffic. Roughly half the tiles fall in category 3, giving causal attention about a **2x speedup** over bidirectional at the same sequence length. FlashAttention-2 with `is_causal=True` and FlashAttention-3 on H100 both implement this skip above diagonal optimization. The other options confuse different concepts: - **Q, K, V shapes are identical** in causal and bidirectional attention, option A is false. - **Cross-head K, V sharing** is MQA/GQA, orthogonal to causal masking, option C is false. - **Softmax is used in both**, option D is false.

TL;DR

FlashAttention skips entire blocks above the causal diagonal because they would softmax to zero, roughly half the work compared to bidirectional, where every block must be computed.

Memory aid

Easy to grasp

Picture filling out a square grid of homework problems. Bidirectional attention is doing every single problem in the grid. Causal attention has a rule: any problem above the diagonal is just zero, do not write anything. A clever student skips those squares entirely instead of writing zero in each one. FlashAttention is that clever student, it inspects the grid tile by tile, sees which tiles are entirely above the diagonal, and skips them without doing the math. Since roughly half the grid is above the diagonal, the clever student does half the work for the same result.

Key concepts

FlashAttention Causal mask GPU kernel optimization

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The fact that causal attention is faster than bidirectional at the same sequence length surprises people. The matrices are the same size; the softmax is the same operation; the value matmul has the same dimensions. The savings live in one place: FlashAttention's tile scheduler skips entire blocks of the score matrix that the causal mask would zero out anyway. This is a hardware-aware optimization that naive implementations miss entirely and that FlashAttention has made the production default since 2023.

This deep dive walks the tile scheduling mechanism in detail, quantifies the speedup, contrasts FlashAttention with naive masked attention, surveys the related optimizations (sliding window, PagedAttention, GQA), and ends with the end to end implications for transformer serving in 2026.

FlashAttention's tile-based processing

Standard attention materializes the full score matrix in high-bandwidth memory:

Compute S = Q K^T of shape (T, T).
Apply mask if causal.
Apply softmax row-wise.
Compute O = A V.

For T = 16,384, the score matrix alone is 2 GB at FP32 (1 GB at FP16). HBM bandwidth becomes the bottleneck.

FlashAttention's insight

Process the score matrix in tiles that fit in SRAM (on-chip cache). For a tile size ts of 64-128, divide both Q and K, V into row blocks of ts. For each output row block, iterate over all key column blocks, computing the partial softmax + value contribution online without materializing the full row.

The key algorithmic trick is the online softmax: maintain a running max and running sum so the partial softmax over each tile composes correctly into the full-row softmax. This is what enables tile by tile processing without losing softmax correctness.

What this gives you

Memory: peak SRAM usage is O(ts^2) instead of O(T^2). For ts = 128 and T = 16,384, that is 16K vs 268M entries, a 16,000x reduction.

Time: the matmul total FLOPs are unchanged, but HBM traffic drops dramatically. Attention becomes compute-bound instead of memory-bound, which on modern GPUs means it actually runs faster.

What changes with the causal mask

The causal mask blocks all entries S[i, j] where j > i. FlashAttention's tile scheduler can inspect tile coordinates and decide which tiles to skip entirely. This is where the causal speedup lives.

FlashAttention is the algorithm. Causal-aware tile skipping is the special case that makes causal attention fast.

Three tile categories

Quantifying the end to end win

Why naive implementations miss the speedup

Related and orthogonal optimizations

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Aspect	Bidirectional	Causal (with FlashAttention)
Q, K, V shapes	(T, d_model)	(T, d_model)
Score matrix tiles computed	All (T/ts)^2	Roughly half (lower triangle + diagonal)
Softmax	Row-wise on full row	Row-wise on causal prefix
Relative kernel cost	1x baseline	~0.5x
Skip mechanism	None	FlashAttention tile scheduler

Real products, models, and research that use this idea.

FlashAttention-2 (Dao 2023): introduced causal-aware tile scheduling; the `is_causal=True` flag activates the upper-triangle skip.
FlashAttention-3 (Shah et al. 2024): brings the same optimization plus warp-specialization to H100 GPUs.

What an interviewer would ask next. Try answering before peeking at the approach.

QIf causal attention is 2x faster on the attention kernel, why is end to end transformer block throughput only 1.3-1.5x faster instead of 2x?

The transformer block also contains the FFN, which is unchanged between causal and bidirectional and is typically 2-3x more compute than attention. So the attention 2x speedup amortizes against a fixed FFN cost. The end to end speedup is bounded by Amdahl's law on the attention/FFN ratio.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Believing causal attention is cheaper because the matrices are smaller. The Q, K, V shapes are identical; the savings come from skipping blocked-out blocks of the score matrix.

60 second bullets to scan on the way to the call.

FlashAttention's tile-based processing of the score matrix
Three tile categories: below, straddling, above the diagonal

Primary sources. Browse if you want the original framing.