Residual Connection
Also known as: Skip connection
Add the input of each sub-layer back to its output: the trick that lets us train 100+ layer transformers.
A skip connection that adds the input of a sub-layer directly to its output. Enables training very deep networks by ensuring gradient flow and identity-mapping capability. Used around every attention and MLP block in transformers.
In practice
If you don't know residuals you can't whiteboard a transformer block. Interview must-have.
Related topics
Questions that mention this term
- Inside a transformer, what is the residual stream and how does attention interact with it?
- Why does a deep…
- After W_O projects the multi-head output back to d_model, what happens before the next sub-layer runs?
- Which components make up a standard transformer block?
- What happens when a token is routed to a null (skip) expert slot in Switch style MoE?
- Name what wraps the attention sub-layer in every transformer block.
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Causal Mask
Attention mask that hides future tokens so each position can only see itself and prior tokens.
Context Window
The max number of tokens a model can attend to at once.
Decoder-Only
Single autoregressive transformer stack: the shape of every modern frontier LLM.
Encoder-Decoder
Transformer with separate encoder + decoder stacks; strong for translation and structured seq2seq tasks.
Layer Normalization
Per-sample, per-position normalization. The glue that keeps transformer training stable.