The one hop gradient path between any two tokens. Parallelism is a consequence, even with infinite parallel RNN hardware, the gradient signal would still degrade across distance.
Think of writing a paragraph while glancing at notes spread across your desk. You can look at any note directly, no matter how far away on the desk it is: one quick glance and back to writing. That's attention. An RNN is like passing a whispered message down a line of people: by the time it reaches the end, half the words are gone. Same task, but the desk of notes setup keeps every detail one quick glance away.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Most readers come to this question knowing the trivia answer, 'transformers replaced RNNs', without having interrogated why. The question separates people who can quote a result from people who can name the architectural primitive that made the result possible.
We'll walk through the RNN gradient story, the attention contrast, why parallelism is downstream rather than causal, and how this single architectural choice cascades into every modern long context model.
The goal isn't to memorize the answer. The goal is to be able to derive it from first principles when an interviewer pushes back.
Why RNN gradients struggle: a Jacobian product
Let's start with the structure of an RNN. The hidden state evolves as h_t = f(W h_{t-1} + U x_t), where f is some non-linearity (tanh, sigmoid, ReLU). To compute how a change at position i affects the loss at position j, the chain rule walks j - i steps backward, multiplying Jacobians at each step.
The gradient between distant positions is a product of d matrices. Bengio et al. 1994 made the consequence precise:
- If the spectral radius of those Jacobians is below 1, the product shrinks geometrically: the vanishing gradient.
- If above 1, the product explodes: also useless.
- The 'goldilocks' regime (spectral radius exactly 1) is a measure-zero set in weight space.
Empirically: vanilla RNNs reliably fail to learn dependencies past ~10–20 tokens. LSTMs and GRUs added gated near identity carry channels (the cell state), pushing the practical horizon to ~200–500 tokens. The product structure is still there: gating just keeps the eigenvalues nearer 1.
This is the invariant the RNN family couldn't escape: the gradient between two tokens is fundamentally a long product, and long products are numerically hostile.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | RNN/LSTM | Attention |
|---|---|---|
| Path length between tokens at distance d | O(d) | O(1) |
| Gradient stability at distance 1000+ | Degrades (vanishes/explodes) | Stable |
| Forward pass parallelism in sequence dim | Sequential | Parallel matmul |
| Compute cost per layer | O(n · d²) | O(n² · d) |
Real products, models, and research that use this idea.
- Bengio et al. 1994 'Learning long term dependencies with gradient descent is difficult' formally characterized the RNN vanishing gradient problem.
- GPT-3's 175B parameters and 2048-token context would have been intractable with LSTMs because of the gradient path, regardless of compute.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf the one hop path is so important, why do modern long context transformers still struggle past ~200K tokens?
Two reasons: (1) softmax attention dilutes weight mass across many tokens as n grows, so the effective signal to noise of long range retrieval degrades. (2) Memory bandwidth and KV cache size scale linearly with context, capping practical usage. The gradient path is fine; the signal sharpness and the memory bound are the new bottlenecks.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Picking the parallelism option (C). It's true but it's the consequence of the one hop property, not the underlying reason.
60 second bullets to scan on the way to the call.
Path length vs gradient stability
Why parallelism is a consequence, not a cause
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.