A 32-layer transformer trains fine with residual connections. A teammate removes them as a 'simplification'. Explain what specifically breaks during training and why.
Free · 2 AI evals / day
**Backpropagation through 32 stacked sublayers means multiplying 32 Jacobian matrices together.** A Jacobian is a matrix of partial derivatives of one sublayer's output with respect to its input. To compute how much the loss depends on the weights of layer 1, the chain rule says you multiply the Jacobian of layer 32, times the Jacobian of layer 31, times the Jacobian of layer 30, and so on, all the way down to layer 1.
If the typical singular value of each Jacobian is slightly less than 1, the product shrinks toward zero as depth grows. That is **vanishing gradient**. If the typical singular value is slightly more than 1, the product grows without bound. That is **exploding gradient**. Either way, the gradient that finally reaches layer 1 is either microscopic (no useful update) or astronomical (NaN in the optimizer). The optimizer can no longer descend.
Residual connections fix this. Each sublayer is wrapped as `x_out = x_in + Sublayer(x_in)`. The Jacobian of this wrapped sublayer is `I + J_sublayer`, where `I` is the identity. The identity path adds 1 to every singular value, which keeps the product bounded. The gradient flows straight back through the identity skip, regardless of how depth-many sublayers are stacked.
**The problem is depth times multiplication.** Backprop through a stack of 32 transformer sublayers (64 if you count attention and FFN separately) is a product of 64 Jacobian matrices. Whether that product behaves at depth depends entirely on the spectral properties of those Jacobians.
For a randomly initialized deep stack without residuals, each sublayer's Jacobian has singular values clustered slightly below 1 (the contractive case) or slightly above 1 (the expansive case). Multiplying 64 of them together pushes the gradient norm exponentially toward zero or toward infinity. Early layers, which sit at the deep end of the gradient chain, get the worst of it. They effectively stop receiving useful gradient signal, so the optimizer cannot fit them. Training stalls or diverges.
**Residual connections change the Jacobian structurally.** When the sublayer is wrapped as `x + Sublayer(x)`, its Jacobian becomes `I + J_sublayer`. The identity contributes 1 to every singular value, anchoring them near 1 regardless of what the sublayer learns. The product of 64 such Jacobians stays well-behaved because each factor is approximately `I + small_delta`, and the product of identity plus small delta matrices grows linearly, not exponentially. The gradient norm at layer 1 stays roughly comparable to the gradient norm at layer 32.
This is not a transformer-specific insight. He et al. proposed residual connections in 2015 for ResNet-style convolutional networks specifically to solve this depth scaling problem; before residuals, 20-plus-layer convnets degraded in accuracy. Vaswani et al. carried the idea into the 2017 transformer recipe with the same motivation. Empirically, removing residuals from a 32-layer transformer makes the loss curve flat: the early layers never train, and the late layers can only fit on top of frozen-random features below them.
Quick reference
TL;DR
Without residuals, the gradient through 32 stacked sublayers is a product of 64 Jacobians; that product vanishes or explodes long before it reaches the early layers.
Think about telling a message down a line of 64 people, where each person is allowed to change the message a little before passing it on. After 64 retellings, the original message is unrecognizable. Now imagine each person is also given a copy of the ORIGINAL message and is only asked to write a small correction next to it. After 64 people, you still have the original plus a stack of small corrections. That second setup is what a residual connection does. During training, the model has to send a correction signal BACK through the line so each person can learn how to do better next time. Without the original-message copy (the residual), that signal gets garbled and the people at the front of the line never hear what they did wrong. With residuals, the signal travels straight back to them.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Very shallow networks (under 6 to 8 layers) At shallow depth, the Jacobian product is short enough that vanishing or exploding gradients are not catastrophic. Residuals still help but are not strictly required.
Architectures with structural identity paths (RNNs with skip connections, state-space models with selective updates) Some architectures already provide a path that preserves gradient flow through depth without explicit residual additions.
Adapter-only fine-tuning on a frozen base model If you're training LoRA or DoRA adapters on a base whose residual paths are already intact, the question of removing residuals does not arise; the base's residual structure is preserved by construction.
Quick scan · 2–4 min
Everything important, quickly.
5 min: chain rule gives a product of 64 Jacobians + product vanishes or explodes + residuals make each Jacobian `I + J` + product behavior switches from exponential to linear + ResNet origin + pre-norm versus post-norm + empirical training collapse without residuals.
Side-by-side
Aspect
Without residuals
With residuals
Effective Jacobian per layer
`J_i` (arbitrary spectrum)
`I + J_i` (anchored near identity)
Gradient norm at depth 32
Exponential in depth (vanish or explode)
Linear in depth
Max trainable depth
~10-15 layers
80+ layers in practice
Sensitivity to initialization
Extreme; needs orthogonal init + LR warmup
Robust; standard init works
What the early layers learn
Stays near random init
Fits the task normally
Real-world examples
Real products, models, and research that use this idea.
1He et al., 'Deep Residual Learning for Image Recognition' (2015), showed a 56-layer plain CNN trained WORSE than a 20-layer one; adding residuals reversed the ordering and was the unlock for very deep convnets.
2Vaswani et al. (2017) imported residual connections from ResNet into the transformer block recipe; every transformer block in production today still uses them.
3Llama 3.1 70B stacks 80 transformer blocks (160 residual connections) and trains stably to convergence on 15T tokens; without residuals this depth would not be reachable.
4Xiong et al. (2020), 'On Layer Normalization in the Transformer Architecture', empirically demonstrated that pre-norm placement (which preserves a clean residual path) is required for stable training past about 20 layers.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does pre-norm specifically help relative to post-norm, given both use residual connections?
Sign in or go Pro to see the approach for this follow-up.
QWhat does the gradient look like at layer 1 of an 80-layer pre-norm transformer at the start of training?
Sign in or go Pro to see the approach for this follow-up.
QIf you wanted to remove residual connections from a transformer, what initialization tricks could you try to make it train?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Calling this 'just' a vanishing-gradient issue. The exploding-gradient case is equally real, and it is the geometric mean of the Jacobian singular values that controls which direction it fails in.