Explain why transformers replaced RNNs for language modeling.

**RNNs** process a sequence one token at a time, carrying a hidden state forward. Token 100 cannot start until tokens 1 through 99 have been processed. That sequential dependency makes training slow and limits how long a context the model can handle. Transformers replace recurrence with **attention**. Every token attends to every other token directly in a single layer, so the path from token 1 to token 100 is one hop instead of 99. That fixes two RNN problems at once: vanishing gradients over long sequences disappear, and the long-range dependencies that RNNs struggled with become trivial to model. The second reason transformers won is hardware. Attention computes all pairwise interactions in parallel as a single matrix multiply, which is exactly what a GPU is built for. RNNs cannot exploit that parallelism because each step depends on the previous one. On the same hardware, a transformer trains roughly an order of magnitude faster than an RNN of comparable size.

**Transformers won on three structural advantages over RNNs.** First, parallelism. An RNN processes tokens strictly in sequence; token `t` cannot start until token `t-1` is done. A transformer processes the entire sequence in one matrix multiply, which lets a modern GPU saturate its tensor cores instead of idling on a sequential dependency. Same hardware, roughly 10x more useful FLOPs per second during training. Second, direct paths. In an RNN, information from token 1 reaches token 100 by being copied through 99 hidden states, with **vanishing gradient** problems along the way. In a transformer, the path length between any two tokens is exactly one attention hop. Long-range dependencies that LSTMs and GRUs struggled with become routine. Third, scaling laws. Because the architecture was parallel and the path length was constant, the field could scale models and data faster than RNNs allowed. The empirical result is that transformers dominate every language benchmark and every modern LLM (GPT, Claude, Llama, Mistral) is a transformer stack.