Residual Connection
Also known as: Skip connection
Add the input of each sub-layer back to its output: the trick that lets us train 100+ layer transformers.
A skip connection that adds the input of a sub-layer directly to its output. Enables training very deep networks by ensuring gradient flow and identity-mapping capability. Used around every attention and MLP block in transformers.
In practice
If you don't know residuals you can't whiteboard a transformer block. Interview must-have.
Related topics
Questions that mention this term
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Transformer
The attention-only neural architecture behind GPT, Claude, Gemini, and almost every modern LLM.
Context Window
The max number of tokens a model can attend to at once.
Mixture of Experts (MoE)
Scale model capacity by routing each token to a small subset of expert MLPs instead of using all of them.
RoPE (Rotary Position Embedding)
Position info injected by rotating Q and K vectors, easy to extend to longer contexts.
Multi-Head Attention (MHA)
Run several attention heads in parallel with different projections, then concat. This captures multiple relationship types per layer.