Why does a deep transformer stack stop training if you remove the residual connections?

Short answer·Medium·4.0 · 0·~3 min·Relevant atAnthropic Mistral AI

Attempt it

A 32-layer transformer trains fine with residual connections. A teammate removes them as a 'simplification'. Explain what specifically breaks during training and why.

Free · 2 AI evals / day

**Backpropagation through 32 stacked sublayers means multiplying 32 Jacobian matrices together.** A Jacobian is a matrix of partial derivatives of one sublayer's output with respect to its input. To compute how much the loss depends on the weights of layer 1, the chain rule says you multiply the Jacobian of layer 32, times the Jacobian of layer 31, times the Jacobian of layer 30, and so on, all the way down to layer 1. If the typical singular value of each Jacobian is slightly less than 1, the product shrinks toward zero as depth grows. That is **vanishing gradient**. If the typical singular value is slightly more than 1, the product grows without bound. That is **exploding gradient**. Either way, the gradient that finally reaches layer 1 is either microscopic (no useful update) or astronomical (NaN in the optimizer). The optimizer can no longer descend. Residual connections fix this. Each sublayer is wrapped as `x_out = x_in + Sublayer(x_in)`. The Jacobian of this wrapped sublayer is `I + J_sublayer`, where `I` is the identity. The identity path adds 1 to every singular value, which keeps the product bounded. The gradient flows straight back through the identity skip, regardless of how depth-many sublayers are stacked.

**The problem is depth times multiplication.** Backprop through a stack of 32 transformer sublayers (64 if you count attention and FFN separately) is a product of 64 Jacobian matrices. Whether that product behaves at depth depends entirely on the spectral properties of those Jacobians. For a randomly initialized deep stack without residuals, each sublayer's Jacobian has singular values clustered slightly below 1 (the contractive case) or slightly above 1 (the expansive case). Multiplying 64 of them together pushes the gradient norm exponentially toward zero or toward infinity. Early layers, which sit at the deep end of the gradient chain, get the worst of it. They effectively stop receiving useful gradient signal, so the optimizer cannot fit them. Training stalls or diverges. **Residual connections change the Jacobian structurally.** When the sublayer is wrapped as `x + Sublayer(x)`, its Jacobian becomes `I + J_sublayer`. The identity contributes 1 to every singular value, anchoring them near 1 regardless of what the sublayer learns. The product of 64 such Jacobians stays well-behaved because each factor is approximately `I + small_delta`, and the product of identity plus small delta matrices grows linearly, not exponentially. The gradient norm at layer 1 stays roughly comparable to the gradient norm at layer 32. This is not a transformer-specific insight. He et al. proposed residual connections in 2015 for ResNet-style convolutional networks specifically to solve this depth scaling problem; before residuals, 20-plus-layer convnets degraded in accuracy. Vaswani et al. carried the idea into the 2017 transformer recipe with the same motivation. Empirically, removing residuals from a 32-layer transformer makes the loss curve flat: the early layers never train, and the late layers can only fit on top of frozen-random features below them.

Quick reference

TL;DR

Without residuals, the gradient through 32 stacked sublayers is a product of 64 Jacobians; that product vanishes or explodes long before it reaches the early layers.

Memory aid

Easy to grasp

Think about telling a message down a line of 64 people, where each person is allowed to change the message a little before passing it on. After 64 retellings, the original message is unrecognizable. Now imagine each person is also given a copy of the ORIGINAL message and is only asked to write a small correction next to it. After 64 people, you still have the original plus a stack of small corrections. That second setup is what a residual connection does. During training, the model has to send a correction signal BACK through the line so each person can learn how to do better next time. Without the original-message copy (the residual), that signal gets garbled and the people at the front of the line never hear what they did wrong. With residuals, the signal travels straight back to them.

Key concepts

Transformer architecture Attention

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: chain rule gives a product of 64 Jacobians + product vanishes or explodes + residuals make each Jacobian `I + J` + product behavior switches from exponential to linear + ResNet origin + pre-norm versus post-norm + empirical training collapse without residuals.

Side-by-side

Aspect	Without residuals	With residuals
Effective Jacobian per layer	`J_i` (arbitrary spectrum)	`I + J_i` (anchored near identity)
Gradient norm at depth 32	Exponential in depth (vanish or explode)	Linear in depth
Max trainable depth	~10-15 layers	80+ layers in practice
Sensitivity to initialization	Extreme; needs orthogonal init + LR warmup	Robust; standard init works
What the early layers learn	Stays near random init	Fits the task normally

Real-world examples

Real products, models, and research that use this idea.

He et al., 'Deep Residual Learning for Image Recognition' (2015), showed a 56-layer plain CNN trained WORSE than a 20-layer one; adding residuals reversed the ordering and was the unlock for very deep convnets.
Vaswani et al. (2017) imported residual connections from ResNet into the transformer block recipe; every transformer block in production today still uses them.
Llama 3.1 70B stacks 80 transformer blocks (160 residual connections) and trains stably to convergence on 15T tokens; without residuals this depth would not be reachable.
Xiong et al. (2020), 'On Layer Normalization in the Transformer Architecture', empirically demonstrated that pre-norm placement (which preserves a clean residual path) is required for stable training past about 20 layers.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does pre-norm specifically help relative to post-norm, given both use residual connections?

QWhat does the gradient look like at layer 1 of an 80-layer pre-norm transformer at the start of training?

QIf you wanted to remove residual connections from a transformer, what initialization tricks could you try to make it train?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Calling this 'just' a vanishing-gradient issue. The exploding-gradient case is equally real, and it is the geometric mean of the Jacobian singular values that controls which direction it fails in.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.