How do linear attention variants (Performer, Linformer) achieve O(n) complexity, and what's the quality cost?

Attempt it

Quick reference

TL;DR

Linear attention swaps softmax for a kernel feature map so (φ(K)ᵀV) can be computed first, making total cost O(n·d²) instead of O(n²·d).

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a party where you want to know how much each guest should pay attention to every other guest. The slow way builds a giant grid of every pair and reads from it; the grid grows fast and is awkward to handle. Linear attention rearranges the order of work. Each guest writes a small summary card about themselves. Combining all the summary cards is cheap and gives a single pocket sized notebook. Each guest then peeks at the notebook instead of the giant grid. The shortcut keeps the work small, but the summary cards round off some detail, so the answers are a touch less precise than the full grid would give.

Key concepts

Linear attention Performer

Quick scan · 2–4 min

Everything important, quickly.

Walk the associativity trick, derive the cost reduction, contrast Performer (kernel approximation) with Linformer (low rank K,V), discuss why softmax is hard to approximate cleanly, note that Mamba is the modern linear time successor via different math.

Side-by-side

Method	Mechanism	Complexity	Approximation?
Standard attention	softmax(QKᵀ) V	O(n²·d)	Exact
FlashAttention	Same math, tiled I/O	O(n²·d) compute, O(n) memory	Exact
Performer	Random feature softmax kernel	O(n·d²)	Yes (unbiased)
Linformer	Low rank K,V projection	O(n·k·d)	Yes (low rank)
Linear Transformer	elu+1 feature map	O(n·d²)	Yes (kernel)
Mamba (SSM)	Selective recurrent state	O(n·d²)	Different math, not attention

Real-world examples

Real products, models, and research that use this idea.

Performer demonstrated linear time attention with FAVOR+ random features.
Linformer low rank projected K, V from n to k=256 for sequences up to 4k.
Linear Transformer used elu(x)+1 as the feature map; one of the earliest linear attention papers.
Reformer used LSH bucketing as a different sparsity flavored linear approximation.
Mamba and Mamba-2 (Gu and Dao, 2023-2024) and hybrid Transformer-SSM stacks shipping by 2026 are the modern linear time successors, distinct math from kernel feature map linear attention.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the softmax kernel exp(q · k) hard to approximate with a separable feature map?

Sign in or go Pro to see the approach for this follow-up.

QLinformer assumes the attention matrix is approximately low rank. When is this true?

Sign in or go Pro to see the approach for this follow-up.

QIs Mamba 'linear attention done right' or something fundamentally different?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.