How do you extend a pretrained LLM's context window beyond its training length using position interpolation / YaRN?

Attempt it

A model was trained with 2048 context but you want to serve 8192. Walk through position interpolation (PI) and how YaRN improves on it. Why is this needed at all?

Free · 2 AI evals / day

Quick reference

TL;DR

PI scales positions uniformly to fit RoPE inside its trained angle range; YaRN does the same per-frequency and adds a softmax temperature fix.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

RoPE positions are like clock hand angles. The model was taught on clock positions 0 through 2000 and has never seen a clock pointing past that. If you ask it to read position 8000, the clock hand sits in totally unfamiliar territory and the model gets confused. Position interpolation says: spin the clock four times slower. Now position 8000 lands at a familiar 'step 2000' clock face. The model recognizes it. YaRN is the smarter version of the same trick. It spins the slow hands slower (those carry long range information that needs the stretch) but leaves the fast hands alone (those tell apart neighbors and you do not want to blur that).

Key concepts

Rotary Position Embedding Position interpolation YaRN

Quick scan · 2–4 min

Everything important, quickly.

Walk why RoPE fails outside training (angles out of distribution), explain PI's uniform position scaling with fine tune, articulate YaRN's frequency band aware approach, mention the softmax temperature fix, and name modern long context model adopters.

Side-by-side

Method	How it works	Quality cost	Fine tune needed
No scaling	Just feed long positions to RoPE	Catastrophic failure outside training	N/A
Position Interpolation	Scale all positions m → m/r linearly	Blurs fine grained local distinctions	100M-1B tokens
NTK-aware scaling	Adjust RoPE base frequency	Less blur than PI	100M-1B tokens
YaRN	Frequency-aware scaling + temperature fix	Best preservation of local detail	10M-100M tokens
LongRoPE	Per-dimension search-based scaling	Even better at extreme ratios	Modest

Real-world examples

Real products, models, and research that use this idea.

Llama 1 7B extended from 2k to 32k via position interpolation was the first deployed PI recipe.
Llama 3.1 8B and 70B extended from 8k to 128k using YaRN-style scaling plus brief fine tune.
Llama 4 Maverick, Mistral Large 3, DeepSeek V4, and Qwen 3 ship long context with YaRN-derived RoPE scaling.
LongRoPE searches per-dimension scaling factors and extends to 2M+ context.
Mistral 7B Instruct v0.2 used theta scaling (base frequency adjustment) as an early NTK-aware variant.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does NTK-aware scaling adjust the base frequency rather than positions directly?

Sign in or go Pro to see the approach for this follow-up.

QWhat does YaRN's softmax temperature correction fix?

Sign in or go Pro to see the approach for this follow-up.

QHow do you evaluate whether a context extension worked beyond just looking at perplexity?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.