Attention Mechanism
Also known as: Self-attention, Scaled dot-product attention
How a model decides which input tokens to weight when computing each output token.
A neural network component that lets models dynamically focus on relevant parts of the input sequence. Scaled dot-product attention computes compatibility scores between queries and keys, then uses those scores to weight values.
In practice
Foundational to every transformer-based model. Interviewers probe this to test whether you understand WHY transformers replaced RNNs: not just the formula but the parallelism and long-range dependency wins.
How it compares
Attention is the mechanism; the transformer is the full architecture built around it.
Comparisons that include Attention Mechanism
Related topics
Practice questions
Questions that mention this term
- Match MLA, GQA and MQA to their mechanisms, cache reduction and quality posture
- Name a 2026 task where an encoder-decoder transformer still beats a decoder-only LLM
- What is the attention mechanism, and what makes it structurally different from convolution and recurrence?
- When does adding more instructions to a prompt start to HURT rather than help?
Related terms
Causal Mask
Attention mask that hides future tokens so each position can only see itself and prior tokens.
Context Window
The max number of tokens a model can attend to at once.
Decoder-Only
Single autoregressive transformer stack: the shape of every modern frontier LLM.
Encoder-Decoder
Transformer with separate encoder + decoder stacks; strong for translation and structured seq2seq tasks.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.