Explain in 2-3 sentences why transformers replaced RNNs for sequence modeling.
Transformers process all tokens in parallel, give every token a one-hop path to every other token, and use GPUs efficiently. RNNs do none of these.
Imagine you are reading a 500-page book and have to summarize it. An RNN is like reading every page in order, never flipping back, while trying to keep the gist of every previous page in your head. By page 400 you have forgotten what happened on page 50. A transformer is like spreading every page out on a giant table and being allowed to look at any page when you need it. Whatever connection a sentence on page 400 has to a sentence on page 50, you can see it directly. The transformer also gets to read many pages at the same time on a parallel computer, while the RNN has to read them strictly one after the other. Faster training, longer memory, better results.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
From 2014 to 2017, the dominant architecture for language was the bidirectional LSTM. By 2019, every state of the art language result was a transformer. That switch is one of the cleanest paradigm shifts in modern machine learning, and understanding why it happened is the difference between knowing transformers as a brand and knowing them as a design.
This question is interview shorthand for 'do you understand the architecture or did you memorize the slogan?' The slogan is parallelism. The architecture is direct paths, GPU fit, and the scaling laws that fell out of both. A good answer hits all three.
The sequential bottleneck inside an RNN
An RNN processes a sequence one step at a time. At each step t, the hidden state h_t depends on the previous state h_{t-1} and the current input x_t. There is no way around this dependency; the math says h_t = f(h_{t-1}, x_t), and the dependency is the entire point of the architecture.
This is fatal on GPU hardware. A GPU has thousands of compute units sitting in parallel waiting for big matrix operations. An RNN gives it small operations stitched together by data dependencies, so the hardware idles most of the time. Even on the best 2017 GPUs, RNN training utilization was often under 30 percent.
The deeper problem. Beyond the wall-clock cost, sequentiality also limited model scale. Doubling parameters on a sequential architecture roughly doubled training time. Doubling parameters on a parallel architecture stays compute-bound, which means you can throw more GPUs at it and get linear speedup. The scaling laws that took LLMs from 100M to 100B parameters are only economical with parallel architectures.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | RNN / LSTM | Transformer |
|---|---|---|
| Sequence processing | Sequential, token by token | Parallel, all tokens at once |
| Path length between any two tokens | O(n) | O(1) per layer |
| GPU utilization | Low (sequential dependency) | High (one big matmul) |
| Compute per layer | O(n * d^2) | O(n^2 * d) + O(n * d^2) |
| Long-range dependencies | Vanishing gradient hurts | Direct attention path |
Real products, models, and research that use this idea.
- OpenAI's GPT-5.5 is a decoder-only transformer stack; no recurrent component, attention everywhere.
- Anthropic's Claude Opus 4.7 is a transformer model with extensions for long context and tool use; the base architecture is the same family.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf transformers are O(n^2) in sequence length, why have they not been replaced by Mamba-style state-space models yet?
Quality at typical context lengths still favors attention, and the ecosystem (FlashAttention, vLLM, paged KV) is mature. SSMs win on cost at extreme length but not on raw quality yet.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying transformers 'have more parameters' as the reason. The architectural choice (parallelism + direct paths) is what enabled the parameter scale, not the other way around.
60 second bullets to scan on the way to the call.
Why RNN sequential dependency hurts GPU utilization
How transformers achieve O(1) path length between any two tokens
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.