Attention is a learned soft retrieval where each token weights every other token directly. Unlike RNN/CNN, every token pair has a one hop gradient path.
Imagine a classroom where every student can instantly ask every other student a question and weight their answers by how relevant they are. A recurrent network is like passing a note around the room one student at a time: by the end the message is garbled. A convolution is like only being allowed to talk to your immediate neighbors. Attention skips all that: any student can reach any other student in one step. That direct reach is what we call attention.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Attention is the central sequence mixing operation of the transformer, and the single architectural choice that unlocked everything we now call modern LLMs. Before 2017, sequence models were dominated by recurrence (LSTMs, GRUs) and convolution (TCNs, ByteNet, WaveNet). Both had hit a ceiling: RNNs couldn't be parallelized and suffered vanishing gradients over long distances, while CNNs needed deep stacks to mix distant tokens and still carried a strong locality bias.
The Vaswani et al. paper introduced attention as the sole sequence mixing primitive, no recurrence, no convolution, and showed it both trained faster and produced better translation quality. The reason wasn't 'matmul is fast', though that helped. The reason was structural: attention gives every pair of tokens a direct, one hop gradient path.
This deep dive walks the mechanism, contrasts it against RNN and CNN at the level of path length and gradient flow, derives the O(n²) tradeoff, and explains why the modern landscape (FlashAttention, sparse attention, linear attention) is essentially three different responses to that one tradeoff.
The mechanism, top to bottom
Project each token's embedding into three vectors via learned linear maps.
The three roles
- Q (query), 'what am I looking for'
- K (key), 'what do I represent / how would I describe myself'
- V (value), 'what content do I contribute when matched'
The full computation is the scaled dot product attention formula:
Reading the formula
The QKᵀ product produces an n×n score matrix where entry (i, j) is how well token i's query matches token j's key. Dividing by √d_k normalizes variance so softmax doesn't saturate. Softmax converts each row to a probability distribution. Multiplying by V takes the weighted sum.
Every output token sees every input token, in a single layer, with weights that are content dependent and learned end to end.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import torch, torch.nn.functional as F
def scaled_dot_product_attention(Q, K, V):
# Q, K, V: (batch, n, d_k)
d_k = Q.size(-1)
scores = Q @ K.transpose(-2, -1) / (d_k ** 0.5)
weights = F.softmax(scores, dim=-1)
return weights @ VReal products, models, and research that use this idea.
- The Vaswani et al. 2017 paper 'Attention Is All You Need' replaced recurrence entirely in the Transformer encoder and decoder.
- GPT-1/2/3/4 and Llama-3 use stacks of causal self-attention layers as their sole sequence mixing primitive.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf attention has O(1) path length between any two tokens, why do we still stack many attention layers in practice?
Single layer attention only mixes via a content based weighted sum; expressing compositional features (the noun phrase that the pronoun refers to is the subject of the verb in clause X) requires multiple rounds of mixing + non-linearity from the MLP blocks. Depth gives compositional reasoning, not range.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying attention 'is parallel matmul', true, but that's a consequence of the architecture, not its defining property. The defining property is the direct pairwise gradient path.
60 second bullets to scan on the way to the call.
One sentence definition of attention
Path length: attention vs RNN vs CNN
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.