RoPE (Rotary Position Embedding)
Also known as: Rotary Position Embedding, Rotary embeddings
Position info injected by rotating Q and K vectors, easy to extend to longer contexts.
A positional encoding scheme that applies position-dependent rotations to query and key vectors in self-attention. Encodes relative position naturally and extends gracefully to longer sequences via scaling or interpolation.
In practice
Used in LLaMA, GPT-NeoX, Qwen, and most modern open LLMs. Senior interviews probe RoPE scaling (NTK, YaRN) for long-context extension.
Related topics
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Transformer
The attention-only neural architecture behind GPT, Claude, Gemini, and almost every modern LLM.
Context Window
The max number of tokens a model can attend to at once.
Mixture of Experts (MoE)
Scale model capacity by routing each token to a small subset of expert MLPs instead of using all of them.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.