Attention lets each token in a sequence directly weight every other token via a learned soft retrieval. Unlike convolution (fixed receptive window) or recurrence (sequential dependency forcing one step at a time updates), attention gives every token pair a direct, one hop gradient path, which is what makes transformers scale and train efficiently.
**Attention** is the central mixing operation inside a transformer. In one sentence: for each token in a sequence, attention computes the output as a weighted sum of every other token, where the weights are learned from data based on how relevant each other token is to the current one.
Mechanically, every token is projected into three vectors, a query Q, a key K, and a value V. For a given token's Q, we compute a similarity score against every other token's K (just a dot product), normalize the scores with softmax to get a probability distribution, then take a weighted sum of the V vectors using those weights. That weighted sum is the output for that position.
What makes it structurally different from convolution or recurrence: a CNN mixes within a fixed local window, so distant tokens only meet after many stacked layers. An RNN passes information one step at a time, so distant tokens are separated by many sequential updates and gradients can vanish. Attention compares every token to every other token directly, any two positions are one hop apart, regardless of distance.
Attention is a learned soft retrieval. Every token projects into Q, K, and V vectors; the Q for one token gets scored against the K of every other token via dot product, you softmax those scores, and the output is a weighted sum over the V vectors.
The formula everyone memorizes is:
$$\text{Attn}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$
Structurally, what makes it different from convolution or recurrence is path length. RNNs have an O(n) sequential path between distant tokens: gradients degrade over that distance. CNNs have O(log n) with stacked dilation. Attention has O(1): every token-pair is one hop, in one layer, with a stable gradient path.
That's the architectural reason transformers replaced RNNs and CNNs for sequence modeling, long range dependencies become learnable in a single layer, and the whole thing parallelizes on GPUs. The tradeoff you pay is O(n²) compute and memory in the attention matrix, which is the entire motivation for FlashAttention and sparse attention variants.
Quick reference
TL;DR
Attention is a learned soft retrieval where each token weights every other token directly. Unlike RNN/CNN, every token pair has a one hop gradient path.
Imagine a classroom where every student can instantly ask every other student a question and weight their answers by how relevant they are. A recurrent network is like passing a note around the room one student at a time: by the end the message is garbled. A convolution is like only being allowed to talk to your immediate neighbors. Attention skips all that: any student can reach any other student in one step. That direct reach is what we call attention.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Very long sequences where n > 100k tokens at training time The O(n²) cost in compute and memory becomes prohibitive; sparse attention patterns (Longformer, BigBird) or linear attention approximations (Performer) trade some of the all to all property for tractable scaling.
Strict streaming inference with sub millisecond per token budgets and tiny n When n is small (n ≤ 32) and latency is dominated by kernel launch overhead, a fused RNN cell can match or beat attention because the constant factors swamp the asymptotic win.
Tasks with strong local only structure and tiny training data (small image classification, narrow context tagging) Attention has no locality prior; a CNN with built in translation equivariance learns much faster in this regime. ViT needed JFT-300M to match ResNet on ImageNet.
Hard real-time control loops on edge devices without attention friendly accelerators FlashAttention kernels assume tensor cores and large SRAM; on a Cortex-M-class MCU the O(n²) constant factor dominates and a depthwise separable conv ships in 1/10th the silicon.
Quick scan · 2–4 min
Everything important, quickly.
Define Q/K/V and the softmax(QKᵀ/√d_k)V formula, contrast O(1) attention path vs O(n) recurrence vs O(log n) convolution, and explain why one hop gradient flow is the architectural reason transformers scaled.
Code example
python
import torch, torch.nn.functional as F
def scaled_dot_product_attention(Q, K, V):
# Q, K, V: (batch, n, d_k)
d_k = Q.size(-1)
scores = Q @ K.transpose(-2, -1) / (d_k ** 0.5)
weights = F.softmax(scores, dim=-1)
return weights @ V
Real-world examples
Real products, models, and research that use this idea.
1The Vaswani et al. 2017 paper 'Attention Is All You Need' replaced recurrence entirely in the Transformer encoder and decoder.
2GPT-1/2/3/4 and Llama-3 use stacks of causal self-attention layers as their sole sequence mixing primitive.
3BERT uses bidirectional self-attention with masked language modeling pretraining.
4Vision Transformers (ViT) demonstrated that the same attention primitive replaces convolution in images when given enough data.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf attention has O(1) path length between any two tokens, why do we still stack many attention layers in practice?
Sign in or go Pro to see the approach for this follow-up.
QIs attention strictly more expressive than convolution, or are they incomparable?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Saying attention 'is parallel matmul', true, but that's a consequence of the architecture, not its defining property. The defining property is the direct pairwise gradient path.