Attention Mechanism
Also known as: Self-attention, Scaled dot-product attention
How a model decides which input tokens to weight when computing each output token.
A neural network component that lets models dynamically focus on relevant parts of the input sequence. Scaled dot-product attention computes compatibility scores between queries and keys, then uses those scores to weight values.
In practice
Foundational to every transformer-based model. Interviewers probe this to test whether you understand WHY transformers replaced RNNs: not just the formula but the parallelism and long-range dependency wins.
How it compares
Attention is the mechanism; the transformer is the full architecture built around it.
Comparisons that include Attention Mechanism
Related topics
Practice questions
Related terms
Transformer
The attention-only neural architecture behind GPT, Claude, Gemini, and almost every modern LLM.
Context Window
The max number of tokens a model can attend to at once.
Mixture of Experts (MoE)
Scale model capacity by routing each token to a small subset of expert MLPs instead of using all of them.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
RoPE (Rotary Position Embedding)
Position info injected by rotating Q and K vectors, easy to extend to longer contexts.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.