Which structural property of attention is the single biggest reason transformers scaled where RNNs did not?

RNNs and transformers both model sequences, but they differ on one structural axis: how information flows between distant tokens. In an RNN, the hidden state at position t is computed from the hidden state at position t-1. So for token 100 to influence the prediction at token 200, its signal passes through 100 sequential weight applications. Each application is a matmul plus a non-linearity. Over that long chain, gradients shrink toward zero (the **vanishing gradient** problem) or blow up. LSTMs added gates to bypass some of the squashing, but the path-length problem persisted. **Attention** is the mechanism where each token directly weights every other token in the sequence and produces a weighted sum. Position 100 and position 200 are one hop apart: not 100. The gradient path from the loss back to any token is short, regardless of distance. That short path is what let transformers train stably on the long sequences where RNNs hit a wall.

RNNs and attention differ on one axis that dwarfs all the others: the **gradient path length** between any two tokens. In an RNN, gradients have to flow through `O(n)` sequential weight applications between distant tokens, which is exactly the vanishing/exploding gradient story Bengio characterized back in 1994. LSTMs and GRUs mitigate it but don't eliminate it. In attention, every Q-K-V triple contributes directly to the loss in one step, so the gradient path is `O(1)` regardless of token distance. The trap on this question is picking parallelism, it's true that attention parallelizes well on GPU, but that's downstream. Even on infinite parallel RNN hardware, the gradient signal would still degrade over long distances. The gradient pathology, not wall clock time, is what kept RNNs from scaling. One hop solves both, and the gradient one is the more fundamental win.