Why did decoder-only architectures take over tasks T5-style encoder-decoders used to win?

**Three things changed between 2020 and 2024.** First, **scale**. A sufficiently large decoder-only model can match a smaller specialized encoder-decoder on the same task, just by being given a task prefix like 'translate to French:'. GPT-3 (175B parameters) demonstrated this in 2020; by GPT-4 era it was overwhelming. Second, **instruction tuning**. Decoder-only models can be trained on a mixture of pretraining data and instruction-following examples using one objective (predict the next token). Encoder-decoders need explicit parallel input/output pairs and a separate cross-attention path. Instruction tuning turned a general purpose decoder-only model into a system that can do any task you ask it to. Third, **simplicity compounds**. One block type is easier to scale, easier to optimize, easier to serve, and uses one KV cache structure for every workload. Encoder-decoders need separate code paths for the encoder, the decoder, and the cross-attention. **Encoder-decoders still win in niches.** Whisper uses the encoder once to process audio, then the decoder generates text while cross-attending to the cached encoder output. That reuse is hard to replicate in a decoder-only model.

**The decoder-only architecture won because scale + instruction tuning made the encoder-decoder split unnecessary for general purpose work.** Three causes. **Cause 1: scale subsumes architectural specialization.** In 2020, T5 base (220M params) was strictly better at translation than GPT-2 (1.5B) was. By 2022, GPT-3 (175B) with a 'translate to French:' prefix matched T5-XXL (11B) on most translation benchmarks. Scale collapsed the architectural advantage. The same pattern played out on summarization, question answering, and code generation. **Cause 2: instruction tuning unifies the objective.** Decoder-only models train on next-token prediction across mixed data. Once you can frame any task as 'instruction: \<task\>, input: \<x\>, output: \<y\>' in a single autoregressive sequence, you can train one model on hundreds of tasks at once. Encoder-decoders require explicit (input, output) parallel pairs at training time, which is more restrictive. **Cause 3: architectural simplicity compounds.** One block type means one inference kernel, one KV cache layout, one serving stack. The KV cache trick that makes decoder-only inference fast at long context is uniform across every workload. Encoder-decoder serving requires separate code paths and the cross-attention KV is shaped differently from self-attention KV. **Where encoder-decoder still wins.** Whisper for speech to text (the encoder reads mel spectrograms once, the decoder reuses that representation across every output token). NLLB for high throughput translation across many language pairs. Any task where the input modality differs structurally from the output, or where the same input drives many output generations, still favors encoder-decoder.