Transformer
The attention-only neural architecture behind GPT, Claude, Gemini, and almost every modern LLM.
The dominant neural architecture for language models, introduced in "Attention Is All You Need" (2017). Replaces recurrence with self-attention, enabling massive parallelism and long range dependency capture.
In practice
Every senior GenAI interview has at least one whiteboard-the-transformer question. Be ready to draw the encoder/decoder blocks and explain residuals + layer norm.
How it compares
The transformer is the full block (attention + MLP + residuals + norm); attention is one component inside it.
Comparisons that include Transformer
Related topics
Questions that mention this term
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Context Window
The max number of tokens a model can attend to at once.
Mixture of Experts (MoE)
Scale model capacity by routing each token to a small subset of expert MLPs instead of using all of them.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
RoPE (Rotary Position Embedding)
Position info injected by rotating Q and K vectors, easy to extend to longer contexts.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.