**RoPE** rotates Q and K by position-dependent angles:
$$\theta_{m, i} = m / \text{base}^{2i / d}$$
The original RoFormer paper used base = 10000. Llama 3 raised it to 500000. Why?
### The slowest band
The largest i has the slowest rotation rate. With base = 10000, that band completes a full cycle every ~62800 positions. With base = 500000, the cycle is ~3.14 million positions.
### Why this matters
If the slowest band wraps around inside the evaluation context, positions on opposite sides of the cycle look IDENTICAL to that band. This is aliasing, and it destroys the model's ability to distinguish those positions.
Llama 3 trains at 8k and runs inference at 128k. With base = 10000, the slowest band wraps around 63k, so positions 100 and 63100 alias. With base = 500000, the cycle is 3.14 million, so positions up to a million stay distinct.
### Why this is the cheapest extension trick
No model surgery, just recompute the cos/sin tables with a different constant. The Q and K projections are unchanged; only the rotation angles differ. A short fine-tune at the new long context locks in the new behavior.
**RoPE's** rotation angle for dimension pair i at position m is `theta_{m, i} = m / base^(2i/d)`. The base parameter spreads the angles across a frequency spectrum.
The slowest-frequency band has the largest i, with rotation rate `~ 1 / base`. With base = 10000 (RoFormer default), that band completes a full cycle every `2 pi * 10000 ~ 62800` positions. With base = 500000 (Llama 3), the cycle stretches to `~ 3.14 million` positions.
### The aliasing problem
If the slowest band wraps around inside the evaluation context, positions on opposite sides of the cycle produce IDENTICAL rotation phases. Attention cannot distinguish them; long-context quality collapses.
Llama 3 was pretrained at 8k but targets 128k inference. With base = 10000, the slowest band aliases inside 63k. With base = 500000, it stays unaliased past 1.5M positions, well past target.
### Why this is the dominant long-context lever
Raising base is a hyperparameter change, not architectural. No new modules, no new parameters. Precompute the cos/sin tables with the new constant; Q and K projections are unchanged.
A short fine-tune at the new long context lets the model adapt to the stretched spectrum. The trade off is very slight short-context regression, intermediate bands also rotate slower, but the cost is small enough that quality stays competitive.
### Where it stops being enough
For extreme context (1M+), raising base alone slows fast bands enough to hurt short-context quality. YaRN's per-band frequency scaling addresses this by stretching only slow bands. Llama 4 and Qwen 3.5 combine high-base RoPE with YaRN-style refinements for native long context.
Quick reference
TL;DR
Larger base stretches all rotation frequencies, especially the slowest band. Distant positions keep producing distinguishable phases past the original training horizon, which extends effective context.
Picture a clock with one super-slow hand that takes a year to complete a full revolution. If you only watch for a month, every day looks distinct (the hand has barely moved). Now imagine slowing that hand down to take ten years per revolution. Now you can watch for years and every day still looks distinct. RoPE has a similar slow hand (the lowest-frequency rotation), and raising the base is how Llama 3 slows it down enough that 128k tokens still produce distinguishable positions instead of looking the same as positions inside the original 4k training range.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Short-context models with fixed context length (e.g., < 4k tokens) If the model never sees positions large enough for aliasing to matter, base tuning is unnecessary. The default base = 10000 works fine. Tuning base for short-context regimes only risks degrading the fast bands.
Models using ALiBi or T5-style relative bias instead of RoPE These schemes do not have a base parameter. Their long-context behavior comes from different mechanisms (ALiBi's linear decay slopes, T5's bucketed bias). Base tuning is a RoPE-specific lever.
When extreme context (1M+) is required and base tuning alone would over-flatten the spectrum At very high base values, the frequency spectrum becomes flat and fast bands lose their natural rotation rate. YaRN or LongRoPE per-band scaling is preferred for 1M+ context because it preserves short-context band rates while extending only the slow bands.
When fine-tuning budget is unavailable Raising base requires a short fine-tune at the new context length to let the model adapt to the stretched spectrum. Without that fine-tune, attention patterns are based on the old spectrum and quality drops. If fine-tuning is impossible, position interpolation (rescale positions at inference) is a cheaper alternative.
Quick scan · 2–4 min
Everything important, quickly.
Write the angle formula, identify the aliasing limit set by the slowest band, explain how raising base stretches the spectrum, note the short-context trade-off, mention YaRN as the per-band refinement.
Real-world examples
Real products, models, and research that use this idea.
1Llama 3 raised base from 10000 to 500000 to support 8k pretraining with 128k inference targets.
2Llama 4 Maverick uses base values around 500000 or higher, paired with continued fine-tuning at native long context.
3Qwen 3.5 uses base tuning combined with YaRN's per-band scaling for clean 1M-token context.
4DeepSeek V4 uses high-base RoPE inside its MLA architecture for long-context inference.
5Early RoFormer experiments used base = 10000, which became the default until long-context demands forced the convention to evolve.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does YaRN improve on uniform base tuning, and when would you choose it over a simple base raise?
Sign in or go Pro to see the approach for this follow-up.
QIf raising base solves long context, why not just always start with base = 10^9?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Thinking larger base makes positions 'closer together' or 'more like training'. It is the opposite: larger base spreads positions further apart in phase space, which is exactly what enables long-context extrapolation.