SwiGLU
Also known as: Swish-Gated Linear Unit
Gated activation in transformer MLP blocks (Swish gate × linear up-projection), used in LLaMA and friends.
An activation function used in the MLP block of modern transformers (LLaMA, PaLM). Combines a Swish gate with a linear up-projection; slightly more expressive than ReLU/GeLU at the cost of one extra projection.
In practice
Small architectural choice with measurable quality gain. Worth knowing for any model-architecture deep-dive question.
Related topics
Questions that mention this term
- Match GPT-2 versus Llama-2 attention design choices to their differences
- Select every architecture choice…
- Which of these are linear layers that 'all linear' LoRA targets in a SwiGLU transformer?
- Pair each 2017 transformer…
- Defend the 4x FFN…
- Pick the recipe a 2026 frontier open weight LLM block actually ships
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Causal Mask
Attention mask that hides future tokens so each position can only see itself and prior tokens.
Context Window
The max number of tokens a model can attend to at once.
Decoder-Only
Single autoregressive transformer stack: the shape of every modern frontier LLM.
Encoder-Decoder
Transformer with separate encoder + decoder stacks; strong for translation and structured seq2seq tasks.
Layer Normalization
Per-sample, per-position normalization. The glue that keeps transformer training stable.