**SDPA**, short for `scaled_dot_product_attention`, is the single attention function PyTorch exposes for modern transformer code. It lives at `torch.nn.functional.scaled_dot_product_attention`.
The trick is that SDPA is not one kernel. It is a dispatcher. Under the hood it can route your call to one of three backends:
- **FlashAttention-2**, when your GPU is Ampere or newer, your dtype is fp16 or bf16, your head dim is supported, and your mask is causal or padding-style.
- **Memory-efficient attention**, an xFormers-style kernel, when FlashAttention's conditions are not met but the tensors still fit a tiled exact path.
- **Naive math attention**, the textbook implementation, as the last-resort fallback for any combination the other two cannot handle.
From your model code's perspective, this is one function call regardless of which backend runs. You no longer need to import `flash_attn` directly to get its speedup; it ships built into PyTorch from 2.0 onward.
**SDPA is a unified attention API** that performs runtime backend dispatch. You call `F.scaled_dot_product_attention(Q, K, V, is_causal=True)` and PyTorch picks the best kernel for your shapes, dtypes, and hardware.
The three backends, in priority order:
- **FlashAttention-2**, the fastest path when supported. Requires Ampere or Hopper, fp16/bf16, head dim divisible by 8 (mostly 64 or 128), and a mask type the kernel understands.
- **Memory-efficient attention**, a tiled exact kernel similar to xFormers. Handles more shapes and dtypes than FlashAttention but is slightly slower.
- **Math attention**, the naive `softmax(QK^T/sqrt(d_k))V` implementation. Always available, used when the others cannot run.
Key property: **the output is the same across backends**, up to floating-point reduction order. SDPA is exact, not approximate, regardless of which path runs.
The practical implication is that model code becomes portable. The same `scaled_dot_product_attention` call runs fast on H100, fast-enough on V100 via the mem-efficient path, and correctly on CPU via the math path. You can also pin a specific backend with the `sdp_kernel` context manager if you need reproducibility.
Quick reference
TL;DR
SDPA is PyTorch's unified attention API. It dispatches at runtime to FlashAttention-2, mem-efficient attention, or a naive math kernel based on shape, dtype, and GPU support.
Picture asking a phone assistant to send a message. You do not pick whether it goes by SMS, iMessage, or email; the assistant looks at who you are messaging and picks the best channel automatically. SDPA is that assistant for one of the slowest steps in a language model. You write one line in your model code and PyTorch quietly checks your GPU, the size of the numbers you are using, and the length of the input, then routes the call to whichever underlying engine will run it fastest. You never have to install a separate library or rewrite your model when a faster engine arrives, because the wrapper picks it up for free.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Custom attention biases like ALiBi or sliding window not natively supported by SDPA's backends The dispatcher falls back to the slow math kernel; importing flash-attn directly or writing a custom Triton kernel will be substantially faster at long context.
Research code that needs exact bitwise reproducibility across PyTorch versions SDPA may pick different backends across versions or hardware, changing the floating-point reduction order; pin the backend with sdp_kernel if reproducibility matters.
Architectures that require access to the full attention matrix during the forward pass FlashAttention-based backends never materialize the n-by-n attention matrix; interpretability tools that scan attention weights need the naive math path explicitly.
Non-standard architectures like MLA (DeepSeek V4) where the attention topology does not fit the standard Q, K, V interface SDPA assumes the textbook QKV signature; custom topologies need their own kernels to get the structural speedup.
Quick scan · 2–4 min
Everything important, quickly.
60s: name SDPA as the unified PyTorch attention API, list the three backends (FlashAttention-2, mem-efficient, math) and selection priority, state output is identical across backends, give the sdp_kernel context manager for backend pinning, mention the typical fallback triggers (unsupported mask, unusual head dim, fp32).
Real-world examples
Real products, models, and research that use this idea.
1PyTorch 2.x ships SDPA as the recommended attention API for all new transformer code, with FlashAttention-2 as the default backend on Ampere and Hopper.
2Hugging Face transformers wires SDPA into Llama, Mistral, Qwen, and Gemma model classes via the attn_implementation='sdpa' flag.
3Llama 4 Maverick and Claude Opus 4.7 training pipelines use SDPA so the same model code runs on H100 (FlashAttention-3 path) and A100 (FlashAttention-2 path) without changes.
4TorchTitan and torchtune training stacks use SDPA exclusively, relying on the dispatcher to keep up with hardware-specific kernel releases.
5DeepSeek V4 inference code uses SDPA for short-context paths and custom kernels for the MLA variant where SDPA's generic dispatch cannot match the topology.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat happens if your model uses a custom attention bias (like ALiBi) that SDPA does not natively support?
Sign in or go Pro to see the approach for this follow-up.
QHow do you debug a case where SDPA is silently picking the slow math backend?
Sign in or go Pro to see the approach for this follow-up.
QWhy does PyTorch ship multiple backends instead of just FlashAttention-2 everywhere?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Believing SDPA is a slower wrapper around FlashAttention. It actually IS FlashAttention-2 when shapes and hardware support it, with mem-efficient and math kernels as automatic fallbacks.