Decoder-Only
Also known as: Decoder only, Causal LM
Single autoregressive transformer stack: the shape of every modern frontier LLM.
A transformer architecture using a single autoregressive stack that consumes input and generates output as one sequence. The dominant LLM shape: GPT, LLaMA, Claude, Gemini, Mistral, Qwen all use it.
In practice
Default architecture for all chat-class models. Knowing why it beats encoder-decoder at scale is a senior architecture interview must.
How it compares
Decoder-only is one stack with causal masking; encoder-decoder has two stacks with cross-attention between them.
Comparisons that include Decoder-Only
Related topics
Questions that mention this term
- Does modern LLM fine-tuning default to causal LM or masked LM, and why?
- Identify the attention side optimization when translating one source sentence into many target languages
- Why can BERT use bidirectional attention but decoder-only LLMs like GPT cannot?
- Explain the difference between self-attention and cross-attention. When does each appear in a real architecture?
- Select pooling strategies that are STANDARD for sentence embeddings (not deprecated)
- Which pooling choice is standard when an embedding model keeps a decoder-only LLM's causal mask?
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Causal Mask
Attention mask that hides future tokens so each position can only see itself and prior tokens.
Context Window
The max number of tokens a model can attend to at once.
Encoder-Decoder
Transformer with separate encoder + decoder stacks; strong for translation and structured seq2seq tasks.
Layer Normalization
Per-sample, per-position normalization. The glue that keeps transformer training stable.
Mixture of Experts (MoE)
Scale model capacity by routing each token to a small subset of expert MLPs instead of using all of them.