Which structural property of attention is the single biggest reason transformers scaled where RNNs did not?

Attempt it

Quick reference

TL;DR

The one hop gradient path between any two tokens. Parallelism is a consequence, even with infinite parallel RNN hardware, the gradient signal would still degrade across distance.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Think of writing a paragraph while glancing at notes spread across your desk. You can look at any note directly, no matter how far away on the desk it is: one quick glance and back to writing. That's attention. An RNN is like passing a whispered message down a line of people: by the time it reaches the end, half the words are gone. Same task, but the desk of notes setup keeps every detail one quick glance away.

Key concepts

Vanishing gradient Transformer architecture

Quick scan · 2–4 min

Everything important, quickly.

Contrast O(d) RNN path with O(1) attention path, explain that the Jacobian product along the RNN chain causes vanishing/exploding gradients, and clarify that parallelism falls out of having no sequential dependency.

Side-by-side

Property	RNN/LSTM	Attention
Path length between tokens at distance d	O(d)	O(1)
Gradient stability at distance 1000+	Degrades (vanishes/explodes)	Stable
Forward pass parallelism in sequence dim	Sequential	Parallel matmul
Compute cost per layer	O(n · d²)	O(n² · d)

Real-world examples

Real products, models, and research that use this idea.

Bengio et al. 1994 'Learning long term dependencies with gradient descent is difficult' formally characterized the RNN vanishing gradient problem.
GPT-3's 175B parameters and 2048-token context would have been intractable with LSTMs because of the gradient path, regardless of compute.
Mamba / S4 explicitly engineer near identity state propagation to recover the long range modeling capability while keeping linear time inference.
The 'Attention Is All You Need' paper's central claim was that the recurrence wasn't needed: the all to all attention path subsumed it.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf the one hop path is so important, why do modern long context transformers still struggle past ~200K tokens?

Sign in or go Pro to see the approach for this follow-up.

QMamba claims linear time inference. How does it preserve long range modeling without an O(n²) attention matrix?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.