Modern decoder-only LLMs like GPT, Llama, and Mistral have no separate encoder and no cross-attention sub-layer. Yet they routinely 'condition on' system prompts, retrieved documents, and chat history. Explain how that conditioning actually happens architecturally, and what the encoder-decoder design buys back at the cost of complexity.
Decoder-only LLMs stuff all context into one causal sequence and let self-attention reach back over the prefix; encoder-decoder pays complexity for reusable encoder K, V across many generations.
Imagine two ways to brief a chef. The encoder-decoder way: a head chef reads the menu and the recipe once, writes a tidy summary card, and hands it to every line cook for the night. The decoder-only way: every single time a line cook starts a new plate, they re-read the menu and recipe themselves, with no summary card. The first way is efficient when many cooks share the same brief. The second way is simpler to set up, and works fine when every plate is for a different customer with different instructions. Modern chatbots use the second way because every conversation is unique anyway.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The shift from encoder-decoder to decoder-only is one of the most consequential architectural choices in modern LLM design. It is also one of the most misunderstood. Junior engineers often hear 'GPT has no cross-attention' and wonder how the model conditions on the system prompt at all. The answer is mechanically simple, has serving-cost implications that dominate production budgets, and explains why the encoder-decoder design has not disappeared even though decoder-only dominates the leaderboards.
This deep dive walks the architectural difference precisely, traces the consequences of choosing one path or the other, explains why prefix caching is the decoder-only analog of cross-attention K, V reuse, and closes with a guide for when each architecture is the right call in 2026.
What in-context conditioning actually means
In a decoder-only model, every input token sits in the same causal sequence as the output tokens. The system prompt is positions 0 to S-1. The retrieved chunks are positions S to S+R-1. The chat history is next. The user's current question follows. Generation begins at position N and proceeds one token at a time.
The attention pattern
Causal self-attention at position t reads K and V at positions 0 through t-1. So when generating the first answer token, the model is attending over every prompt and history token. There is no special handling, no boundary, no separate code path for 'prompt' versus 'answer'. The architecture is uniform across the whole sequence.
Why this surprises people
The encoder-decoder design has explicit names for everything: 'this is the encoder, this is the source, this is the decoder, this is cross-attention reading from encoder output'. Decoder-only collapses all of those into one stream, so the names disappear. People then look for a separate 'context input' and don't find one.
The mental model that helps: in decoder-only, the prompt and the answer are the same kind of object. Both are tokens in the sequence; only their position differs.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | Decoder-only | Encoder-decoder |
|---|---|---|
| Sub-layers per decoder block | 2 (self-attn + FFN) | 3 (self-attn + cross-attn + FFN) |
| Where context lives | Prefix of the causal sequence | Encoder output, read via cross-attn |
| KV caches needed | One (decoder self-attn) | Two (self-attn + cross-attn K, V) |
| Source reuse across requests | Prefix caching (exact-match) | Native, encoder runs once |
| Typical use | General chat, completion | Translation, summarization, ASR |
Real products, models, and research that use this idea.
- Llama 4 Maverick: decoder-only, every system prompt + RAG context shares one causal sequence with the generation.
- GPT-5.5 chat: decoder-only with vLLM-style prefix caching to amortize repeated system-prompt prefixes across requests.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is prefix caching cheaper than re-running cross-attention in encoder-decoder, even though both reuse precomputed K, V?
Cross-attention requires a separate set of W_K, W_V matrices and a second attention pass per decoder block. Prefix caching reuses the already-computed self-attention K, V with zero extra compute beyond the lookup. The structural win of cross-attention is that the encoder representations can be shaped differently from the decoder's; prefix caching is constrained to share the decoder's own representation space.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying decoder-only models 'have no context conditioning' because they lack cross-attention. Conditioning happens through the causal self-attention reading the prefix tokens.
60 second bullets to scan on the way to the call.
Sub-layer count: 2 in decoder-only block vs 3 in encoder-decoder decoder block
How context enters a decoder-only model architecturally
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.