Order the operations inside a single pre-norm transformer block forward pass · Zenaique

Attempt it

1Add the FFN output back onto the residual stream
2Add the attention output back onto the residual stream
3Run the position-wise feed-forward network on the normalized vector
4Apply LayerNorm (or RMSNorm) again to the updated residual stream
5Run multi-head attention on the normalized vector
6Apply LayerNorm (or RMSNorm) to the residual stream x

Quick reference

TL;DR

Pre-norm block: norm, attention, add residual, norm again, FFN, add residual. The pattern (norm -> sublayer -> add) repeats twice per block.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a sandwich assembly line where bread comes through twice. The first time it arrives, a quality inspector measures it and rescales it (that's the norm), then a chef adds the filling (attention), then the sandwich is reassembled onto the original bread (the residual add). Then the whole thing goes through a second station: another inspector rescales the new sandwich, another chef adds a different filling (the feed-forward network), and another reassembly step. Pre-norm means the inspector comes BEFORE the chef both times. That ordering matters enormously: it lets you stack the assembly line a hundred stations deep without the bread getting soggy. The original 2017 paper put the inspector after the chef, and that version stops working past about 24 stations.

Key concepts

Transformer architecture Attention

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: list the six steps in order, contrast with post-norm, explain why the residual is an identity path, and mention the final norm before the LM head.

Code example

python

class PreNormBlock(nn.Module):
    def __init__(self, d_model):
        self.norm1 = RMSNorm(d_model)
        self.attn  = GroupedQueryAttention(d_model)
        self.norm2 = RMSNorm(d_model)
        self.ffn   = SwiGLU_FFN(d_model)

    def forward(self, x):
        # Step 1-3: attention sublayer
        x = x + self.attn(self.norm1(x))
        # Step 4-6: FFN sublayer
        x = x + self.ffn(self.norm2(x))
        return x

Real-world examples

Real products, models, and research that use this idea.

Llama 4 Maverick, Llama 3.1, Mistral Large 3, and Qwen 3.5 all use the pre-norm order exactly as listed.
GPT-2 (2019) was the first widely used pre-norm transformer; every frontier model since has followed.
The original BERT (2018) used post-norm; the practical training tricks (warmup, careful learning rate) were necessary because of that choice.
Open source reference implementations like nanoGPT and Hugging Face's `LlamaDecoderLayer` show the six step pre-norm structure explicitly.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat changes in the six step order if you switch to post-norm?

Sign in or go Pro to see the approach for this follow-up.

QWhy is there an extra norm at the very end of a pre-norm stack but not a post-norm stack?

Sign in or go Pro to see the approach for this follow-up.

QIf you fuse the norm into the sublayer call, does the block code change?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Putting the norm after the residual add (that's post-norm, the original 2017 design). Or forgetting that the norm runs twice per block: once before attention, once before FFN.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.