After W_O projects the multi-head output back to d_model, what happens before the next sub-layer runs?

After multi-head attention concatenates per-head outputs and passes them through W_O, the result is added back into the **residual stream**. The full formula for the attention sub-block in a pre-norm transformer is: $$x_\text{out} = x_\text{in} + W_O \cdot \text{Concat}(\text{head}_1, \ldots, \text{head}_h)$$ The addition matters. Without it, each layer would replace the previous layer's output, and information from early layers would be lost by the time the model is 10 or 20 layers deep. With the addition, each layer contributes a delta on top of what is already there, so the embedding signal from layer 0 is still readable at layer 80. The addition also helps gradients. During training, gradients flow back through the skip path without passing through every attention operation, so they reach early layers with reasonable magnitude. Without skip connections, deep transformers do not train, the gradients vanish or explode. This is why the wrong answers fail: replacement breaks the residual property, concatenation doubles the feature width every block (which the architecture is not designed for), and multiplicative gating is a different mechanism used by GLU and Mamba layers.

The attention sub-block writes to the **residual stream** by addition, not replacement. The sequence inside a pre-norm transformer block is: apply LayerNorm, run multi-head attention, project the concatenated heads through W_O, then add the result to the unnormalized input: $$x_\text{out} = x_\text{in} + W_O \cdot \text{Concat}(\text{head}_1, \ldots, \text{head}_h)$$ The addition is the defining design choice that makes deep transformer training tractable. Two things flow from it. First, the residual stream serves as a shared bus that every layer reads from and writes to. Information from the embedding layer is preserved up through the final layer because nothing overwrites the stream, layers only add to it. Second, gradients during the backward pass have a direct path back to early layers through the skip connection, which means they do not vanish even at 80 or 100 layer depths. Replacement would break both properties. Information would be lost layer by layer, and gradients would have to flow back through every attention computation, which is unstable at depth. Concatenation would double the feature width on every block. After 10 blocks the model would be operating on a vector 1024 times wider than the embedding dimension. The architecture is not designed for this and it would explode compute and memory. Multiplicative gating is the GLU and Mamba pattern. Standard transformer attention uses additive residuals. Both designs work but they are different commitments.