SwiGLU
Also known as: Swish-Gated Linear Unit
Gated activation in transformer MLP blocks (Swish gate × linear up-projection), used in LLaMA and friends.
An activation function used in the MLP block of modern transformers (LLaMA, PaLM). Combines a Swish gate with a linear up-projection; slightly more expressive than ReLU/GeLU at the cost of one extra projection.
In practice
Small architectural choice with measurable quality gain. Worth knowing for any model-architecture deep dive question.
Related topics
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Transformer
The attention-only neural architecture behind GPT, Claude, Gemini, and almost every modern LLM.
Context Window
The max number of tokens a model can attend to at once.
Mixture of Experts (MoE)
Scale model capacity by routing each token to a small subset of expert MLPs instead of using all of them.
RoPE (Rotary Position Embedding)
Position info injected by rotating Q and K vectors, easy to extend to longer contexts.
Multi-Head Attention (MHA)
Run several attention heads in parallel with different projections, then concat. This captures multiple relationship types per layer.