Transformer
The attention-only neural architecture behind GPT, Claude, Gemini, and almost every modern LLM.
The dominant neural architecture for language models, introduced in "Attention Is All You Need" (2017). Replaces recurrence with self-attention, enabling massive parallelism and long range dependency capture.
In practice
Every senior GenAI interview has at least one whiteboard-the-transformer question. Be ready to draw the encoder/decoder blocks and explain residuals + layer norm.
How it compares
The transformer is the full block (attention + MLP + residuals + norm); attention is one component inside it.
Comparisons that include Transformer
Related topics
Questions that mention this term
- Which part of a…
- Walk through a T5 decoder block in order, list the attention sub-layers and what each one's Q, K, V read from.
- Spot the masking bug in this packed sequence training setup
- Walk through how a token's d_model embedding is split into per head pieces inside multi-head attention.
- Why did decoder-only architectures…
- Identify the context vector inside scaled dot product attention and what produces it.
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Causal Mask
Attention mask that hides future tokens so each position can only see itself and prior tokens.
Context Window
The max number of tokens a model can attend to at once.
Decoder-Only
Single autoregressive transformer stack: the shape of every modern frontier LLM.
Encoder-Decoder
Transformer with separate encoder + decoder stacks; strong for translation and structured seq2seq tasks.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.