Spot the error in this transformer block that silently dropped its residuals

The block has two sublayers (attention and FFN), and **each one is missing its residual addition**. The correct shape is: ``` def block(x): a = x + attn(norm1(x)) x = a + ffn(norm2(a)) return x ``` Look at the original. Line 2 says `a = attn(norm1(x))`. That throws away `x` and keeps only the attention output. Line 3 says `x = ffn(norm2(a))`. That throws away `a` and keeps only the FFN output. Neither line preserves the running stream. Why this matters. The **residual connection** is the additive skip path that makes deep transformers trainable. It does two things at once. First, it gives the gradient a direct path back through every layer (the gradient of `x + f(x)` with respect to x is `1 + f'(x)`, so even if f's gradient vanishes, the 1 keeps gradients alive). Second, it gives each sublayer the property of being an UPDATE to the stream rather than a REPLACEMENT, which is what the interpretability and stability theory of transformers is built on. Delete both residuals and the block trains for the first few layers, then stalls. A 6-layer model might still work poorly. A 64-layer model will not learn at all.

**Both residuals are missing.** The pre-norm transformer block is: $$x \leftarrow x + \text{Attn}(\text{Norm}_1(x)), \quad x \leftarrow x + \text{FFN}(\text{Norm}_2(x))$$ The buggy code performs both sublayer computations but throws away the original residual, replacing `x` with the sublayer output instead of adding to it. The fix: ``` def block(x): a = x + attn(norm1(x)) x = a + ffn(norm2(a)) return x ``` Why both additions matter. The residual connection has two distinct roles. First, the **gradient highway**: the derivative of `x + f(x)` is `1 + f'(x)`, so gradients always have a direct backpropagation path that does not depend on f's local gradient. This is what lets you train 64+ layer stacks; without it, the gradient vanishes geometrically with depth. Second, the **shared residual stream**: every sublayer in the network reads from and writes to this stream additively. Interpretability work treats the stream as a workspace for human-meaningful features. Overwriting it makes every layer's output the only signal forward, which collapses the rich layered representation transformers depend on. Downstream symptoms of this bug. Training loss starts high and barely decreases. Gradient norms at the bottom layers are tiny. Even shallow models (6-12 layers) learn poorly. Adding the residuals back instantly recovers normal training behavior, which is the cleanest diagnostic if you suspect this bug in a custom model implementation.