Identify what changes when a block computes attention and FFN in parallel, PaLM style

**The standard block is sequential.** It reads the residual stream, normalizes it, runs attention, adds the result back into the stream, then normalizes again, runs the FFN, and adds that. Two add steps, two norm steps, run one after the other. A **parallel block** instead runs attention and FFN from the SAME normalized input, side by side, and adds both outputs into the residual in one shot: $$y = x + \text{attn}(\text{norm}(x)) + \text{ffn}(\text{norm}(x))$$ The attention sublayer and the FFN sublayer are now siblings, not a chain. They can run concurrently on the hardware, and because they read the same input, their input projection matrices can be fused into a single wider matmul that the GPU executes more efficiently. The trade is that the FFN no longer sees the current layer's attention output. In a sequential block, the FFN at layer L gets the attention update from layer L baked in. In a parallel block, the FFN at layer L only sees what the residual stream looked like before this layer's attention fired. PaLM reported the speedup is about 15 percent at very large scale with negligible quality loss, but small models lose a bit of quality from the missing coupling. GPT-J, PaLM, and Falcon use parallel blocks. Llama and Mistral stayed sequential.

**Parallel blocks** compute attention and FFN from the same pre-norm input and add both back to the residual stream in one step. The block equation goes from sequential `y = x + ffn(norm(x + attn(norm(x))))` to parallel $$y = x + \text{attn}(\text{norm}(x)) + \text{ffn}(\text{norm}(x))$$ **Why the hardware wins.** Both sublayers read `norm(x)`. Their input projections (`W_Q`, `W_K`, `W_V` for attention; the FFN up-projection) can be concatenated into one wide matmul, which is more efficient than two narrower ones. The attention and FFN bodies then run as independent CUDA streams. PaLM reported a ~15 percent training step speedup at 540B scale; the gain is largest where matmul saturates compute. **What you give up.** In the sequential form, the FFN's input contains this layer's attention update. In the parallel form, it does not; the FFN reads the pre-attention residual. The two sublayers can still cooperate, but only across layers, not within one. Small models (sub-7B) measurably lose quality from this; at frontier scale the loss disappears in the noise. **Who ships it.** GPT-J, PaLM, Falcon. Llama 1/2/3, Mistral, DeepSeek, Qwen stayed sequential because the simpler chained form is easier to tune and the speedup matters less at 7B-70B scales than it did at 540B. The 2026 convention for new frontier decoders is sequential pre-norm by default, parallel only when training-compute pressure is severe.