Llama-3 bumps RoPE base from 10k to 500k, which long context property follows?

MCQ·Medium·4.0 · 0·~1 min·Asked atModal Labs Qdrant Workday·Relevant atMeta Microsoft

Attempt it

**RoPE** rotates Q and K by position-dependent angles: $$\theta_{m, i} = m / \text{base}^{2i / d}$$ The original RoFormer paper used base = 10000. Llama 3 raised it to 500000. Why? ### The slowest band The largest i has the slowest rotation rate. With base = 10000, that band completes a full cycle every ~62800 positions. With base = 500000, the cycle is ~3.14 million positions. ### Why this matters If the slowest band wraps around inside the evaluation context, positions on opposite sides of the cycle look IDENTICAL to that band. This is aliasing, and it destroys the model's ability to distinguish those positions. Llama 3 trains at 8k and runs inference at 128k. With base = 10000, the slowest band wraps around 63k, so positions 100 and 63100 alias. With base = 500000, the cycle is 3.14 million, so positions up to a million stay distinct. ### Why this is the cheapest extension trick No model surgery, just recompute the cos/sin tables with a different constant. The Q and K projections are unchanged; only the rotation angles differ. A short fine-tune at the new long context locks in the new behavior.

**RoPE's** rotation angle for dimension pair i at position m is `theta_{m, i} = m / base^(2i/d)`. The base parameter spreads the angles across a frequency spectrum. The slowest-frequency band has the largest i, with rotation rate `~ 1 / base`. With base = 10000 (RoFormer default), that band completes a full cycle every `2 pi * 10000 ~ 62800` positions. With base = 500000 (Llama 3), the cycle stretches to `~ 3.14 million` positions. ### The aliasing problem If the slowest band wraps around inside the evaluation context, positions on opposite sides of the cycle produce IDENTICAL rotation phases. Attention cannot distinguish them; long-context quality collapses. Llama 3 was pretrained at 8k but targets 128k inference. With base = 10000, the slowest band aliases inside 63k. With base = 500000, it stays unaliased past 1.5M positions, well past target. ### Why this is the dominant long-context lever Raising base is a hyperparameter change, not architectural. No new modules, no new parameters. Precompute the cos/sin tables with the new constant; Q and K projections are unchanged. A short fine-tune at the new long context lets the model adapt to the stretched spectrum. The trade off is very slight short-context regression, intermediate bands also rotate slower, but the cost is small enough that quality stays competitive. ### Where it stops being enough For extreme context (1M+), raising base alone slows fast bands enough to hurt short-context quality. YaRN's per-band frequency scaling addresses this by stretching only slow bands. Llama 4 and Qwen 3.5 combine high-base RoPE with YaRN-style refinements for native long context.

TL;DR

Larger base stretches all rotation frequencies, especially the slowest band. Distant positions keep producing distinguishable phases past the original training horizon, which extends effective context.

Memory aid

Easy to grasp

Picture a clock with one super-slow hand that takes a year to complete a full revolution. If you only watch for a month, every day looks distinct (the hand has barely moved). Now imagine slowing that hand down to take ten years per revolution. Now you can watch for years and every day still looks distinct. RoPE has a similar slow hand (the lowest-frequency rotation), and raising the base is how Llama 3 slows it down enough that 128k tokens still produce distinguishable positions instead of looking the same as positions inside the original 4k training range.

Key concepts

RoPE Long-context extension YaRN

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Llama 3's decision to raise RoPE's base from 10000 to 500000 is one of the most consequential 'small' hyperparameter choices in modern LLM architecture. It is the difference between a model with a hard 31k effective context aliasing wall and a model that can serve 128k natively with no architectural changes. Understanding why the base parameter does this, and what the trade-offs are, is essential for reasoning about long-context model design in 2026.

This deep dive starts with the RoPE angle formula, explains the frequency spectrum and the aliasing limit, derives why the slowest band sets the long-context wall, walks through what changes when base is raised from 10000 to 500000, covers the short-context quality trade-off, and closes with YaRN as the per-band refinement that pushes context to 1M+.

By the end you should be able to compute the aliasing limit for any base value, predict the short-context cost of a base raise, and explain when YaRN is needed instead of (or in addition to) simple base tuning.

The angle formula and the frequency spectrum

RoPE rotates Q and K vectors by position-dependent angles. The angle for dimension pair i at token position m is:

\theta_{m, i} = m / \text{base}^{2i / d}

where d is the per-head dimension, i ranges from 0 to d/2 - 1, and base is the hyperparameter.

Decoding the formula

For a fixed i, the angle grows LINEARLY with m, so the rotation accumulates with position. For a fixed m, the angle SHRINKS as i grows (higher i = slower rotation rate per position).

The two extremes

i = 0 (fastest band): rate is 1 / base^0 = 1. Full cycle every 2 pi ~ 6.28 positions. INVARIANT to base. The fastest band is always the same regardless of base.
i = d/2 - 1 (slowest band): rate is roughly 1 / base^((d - 2) / d) ~ 1 / base. Full cycle every 2 pi * base positions. SCALES LINEARLY with base.

Intermediate bands interpolate geometrically between these two extremes. The spectrum is a geometric progression of rotation rates from fast to slow.

Why a spectrum and not a single rate

A single rotation rate would be either too fast (loses long-distance distinguishability) or too slow (loses short-distance resolution). The geometric spectrum gives multiscale resolution: fast bands encode short-range relative patterns, slow bands encode long-range ones. This is the same multiscale principle behind multi-resolution analysis in signal processing.

\theta_{m, i} = m / \text{base}^{2i / d}

Why the slowest band sets the aliasing wall

What actually changes when you raise base

The short-context trade off

The 2026 long-context recipe

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Llama 3 raised base from 10000 to 500000 to support 8k pretraining with 128k inference targets.
Llama 4 Maverick uses base values around 500000 or higher, paired with continued fine-tuning at native long context.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does YaRN improve on uniform base tuning, and when would you choose it over a simple base raise?

YaRN scales different frequency bands non-uniformly: fast bands get little scaling, slow bands get aggressive scaling. This preserves short-context quality where the fast bands carry most of the signal, while extending long context via the slow bands. Choose YaRN when target context is extreme (1M+) or when short-context quality regression from base tuning alone is unacceptable.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Thinking larger base makes positions 'closer together' or 'more like training'. It is the opposite: larger base spreads positions further apart in phase space, which is exactly what enables long-context extrapolation.

60 second bullets to scan on the way to the call.

Write the RoPE angle formula
Identify which dimension band sets the aliasing limit

Primary sources. Browse if you want the original framing.