Llama-3 bumps RoPE base from 10k to 500k, which long-context property follows?

MCQ·Medium·4.0 · 0·~1 min·Relevant atMeta Microsoft

Attempt it

**RoPE** rotates Q and K by position-dependent angles: $$\theta_{m, i} = m / \text{base}^{2i / d}$$ The original RoFormer paper used base = 10000. Llama 3 raised it to 500000. Why? ### The slowest band The largest i has the slowest rotation rate. With base = 10000, that band completes a full cycle every ~62800 positions. With base = 500000, the cycle is ~3.14 million positions. ### Why this matters If the slowest band wraps around inside the evaluation context, positions on opposite sides of the cycle look IDENTICAL to that band. This is aliasing, and it destroys the model's ability to distinguish those positions. Llama 3 trains at 8k and runs inference at 128k. With base = 10000, the slowest band wraps around 63k, so positions 100 and 63100 alias. With base = 500000, the cycle is 3.14 million, so positions up to a million stay distinct. ### Why this is the cheapest extension trick No model surgery, just recompute the cos/sin tables with a different constant. The Q and K projections are unchanged; only the rotation angles differ. A short fine-tune at the new long context locks in the new behavior.

**RoPE's** rotation angle for dimension pair i at position m is `theta_{m, i} = m / base^(2i/d)`. The base parameter spreads the angles across a frequency spectrum. The slowest-frequency band has the largest i, with rotation rate `~ 1 / base`. With base = 10000 (RoFormer default), that band completes a full cycle every `2 pi * 10000 ~ 62800` positions. With base = 500000 (Llama 3), the cycle stretches to `~ 3.14 million` positions. ### The aliasing problem If the slowest band wraps around inside the evaluation context, positions on opposite sides of the cycle produce IDENTICAL rotation phases. Attention cannot distinguish them; long-context quality collapses. Llama 3 was pretrained at 8k but targets 128k inference. With base = 10000, the slowest band aliases inside 63k. With base = 500000, it stays unaliased past 1.5M positions, well past target. ### Why this is the dominant long-context lever Raising base is a hyperparameter change, not architectural. No new modules, no new parameters. Precompute the cos/sin tables with the new constant; Q and K projections are unchanged. A short fine-tune at the new long context lets the model adapt to the stretched spectrum. The trade off is very slight short-context regression, intermediate bands also rotate slower, but the cost is small enough that quality stays competitive. ### Where it stops being enough For extreme context (1M+), raising base alone slows fast bands enough to hurt short-context quality. YaRN's per-band frequency scaling addresses this by stretching only slow bands. Llama 4 and Qwen 3.5 combine high-base RoPE with YaRN-style refinements for native long context.

Quick reference

TL;DR

Larger base stretches all rotation frequencies, especially the slowest band. Distant positions keep producing distinguishable phases past the original training horizon, which extends effective context.

Memory aid

Easy to grasp

Picture a clock with one super-slow hand that takes a year to complete a full revolution. If you only watch for a month, every day looks distinct (the hand has barely moved). Now imagine slowing that hand down to take ten years per revolution. Now you can watch for years and every day still looks distinct. RoPE has a similar slow hand (the lowest-frequency rotation), and raising the base is how Llama 3 slows it down enough that 128k tokens still produce distinguishable positions instead of looking the same as positions inside the original 4k training range.

Key concepts

RoPE Long-context extension YaRN

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Write the angle formula, identify the aliasing limit set by the slowest band, explain how raising base stretches the spectrum, note the short-context trade-off, mention YaRN as the per-band refinement.

Real-world examples

Real products, models, and research that use this idea.

Llama 3 raised base from 10000 to 500000 to support 8k pretraining with 128k inference targets.
Llama 4 Maverick uses base values around 500000 or higher, paired with continued fine-tuning at native long context.
Qwen 3.5 uses base tuning combined with YaRN's per-band scaling for clean 1M-token context.
DeepSeek V4 uses high-base RoPE inside its MLA architecture for long-context inference.
Early RoFormer experiments used base = 10000, which became the default until long-context demands forced the convention to evolve.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does YaRN improve on uniform base tuning, and when would you choose it over a simple base raise?

QIf raising base solves long context, why not just always start with base = 10^9?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Thinking larger base makes positions 'closer together' or 'more like training'. It is the opposite: larger base spreads positions further apart in phase space, which is exactly what enables long-context extrapolation.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.