Decoder-only LLMs ship without a cross-attention sub-layer, so where does the context go?

A decoder-only transformer like Llama or Mistral has one stack of identical blocks. Each block runs causal self-attention then a feed-forward layer. There is no separate encoder, and no cross-attention sub-layer at all. So where does the context go? Into the same input sequence as the generation. The system prompt, the retrieved chunks, the prior chat turns, and the user's question are concatenated into one long token stream. The model then generates the next token by causal self-attention reaching back over every token in that stream. This is called **in-context conditioning**. There is no architectural separation between 'the prompt' and 'the answer',they are different positions in the same sequence. What does encoder-decoder give up by adding cross-attention back? Mostly simplicity. It earns one thing: an encoder output is fixed for a given source, so you can compute it once and reuse it across many decoder runs. Decoder-only has to walk through the prompt tokens again for every new request, though prefix caching helps within a session.

Decoder-only architectures collapse context and generation into one causal sequence. The prompt, retrieved docs, chat history, and user turn are all concatenated; causal self-attention then reads back over the whole prefix as it generates each next token. There is no separate encoder pass and no cross-attention sub-layer. That simplicity is the win. Half as many sub-layers per block, one positional scheme, one KV cache, one residual stream. It is why decoder-only swept the field after GPT-2. Encoder-decoder buys back one real thing: a fixed encoder output that can be reused. Machine translation serving the same English sentence to French, Spanish, and German runs the encoder once and lets all three decoder runs read the same cached cross-attention K, V. Decoder-only has to reprocess the source as a prefix every time. **Prefix caching** is decoder-only's partial answer, vLLM and SGLang reuse the prefix KV across requests that share leading tokens. But it requires exact prefix match and dies the moment branches diverge. For workloads dominated by shared-source many-target patterns the encoder-decoder is structurally more efficient; for general chat with unique inputs each turn, decoder-only wins on simplicity and KV pressure.