Decoder-Only
Also known as: Decoder only, Causal LM
Single autoregressive transformer stack: the shape of every modern frontier LLM.
A transformer architecture using a single autoregressive stack that consumes input and generates output as one sequence. The dominant LLM shape: GPT, LLaMA, Claude, Gemini, Mistral, Qwen all use it.
In practice
Default architecture for all chat-class models. Knowing why it beats encoder-decoder at scale is a senior architecture interview must.
How it compares
Decoder-only is one stack with causal masking; encoder-decoder has two stacks with cross-attention between them.
Comparisons that include Decoder-Only
Related topics
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Transformer
The attention-only neural architecture behind GPT, Claude, Gemini, and almost every modern LLM.
Context Window
The max number of tokens a model can attend to at once.
Mixture of Experts (MoE)
Scale model capacity by routing each token to a small subset of expert MLPs instead of using all of them.
RoPE (Rotary Position Embedding)
Position info injected by rotating Q and K vectors, easy to extend to longer contexts.
Multi-Head Attention (MHA)
Run several attention heads in parallel with different projections, then concat. This captures multiple relationship types per layer.