Decoder-Only
Also known as: Decoder only, Causal LM
Single autoregressive transformer stack: the shape of every modern frontier LLM.
A transformer architecture using a single autoregressive stack that consumes input and generates output as one sequence. The dominant LLM shape: GPT, LLaMA, Claude, Gemini, Mistral, Qwen all use it.
In practice
Default architecture for all chat-class models. Knowing why it beats encoder-decoder at scale is a senior architecture interview must.
How it compares
Decoder-only is one stack with causal masking; encoder-decoder has two stacks with cross-attention between them.
Comparisons that include Decoder-Only
Related topics
Questions that mention this term
- Why did decoder-only architectures…
- Translating one source sentence into many target languages, what attention side trick scales?
- Decoder-only LLMs ship without a cross-attention sub-layer, so where does the context go?
- Explain when an architecture uses cross-attention vs self-attention. Give concrete examples.
- Modern LLM fine-tuning defaults to causal LM or masked LM, pick one and say why
- Pick the only sound pooling choice when an embedding model is built from a decoder-only LLM
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Causal Mask
Attention mask that hides future tokens so each position can only see itself and prior tokens.
Context Window
The max number of tokens a model can attend to at once.
Encoder-Decoder
Transformer with separate encoder + decoder stacks; strong for translation and structured seq2seq tasks.
Layer Normalization
Per-sample, per-position normalization. The glue that keeps transformer training stable.
Mixture of Experts (MoE)
Scale model capacity by routing each token to a small subset of expert MLPs instead of using all of them.