Why does pure long-context FT (without RoPE scaling) fail to extrapolate?

Attempt it

Quick reference

TL;DR

RoPE rotation angles past the trained context length are off-distribution. The model never learned them, so fine-tuning alone cannot help until you rescale the frequencies.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a clock where each word's position is an angle on the dial. The model spent all its training watching the hand sweep from 0 to 4 o'clock, so it only knows what those angles mean. Now you ask it to read positions way out at 32 o'clock. The hand spins to an angle it has literally never seen, and the model has no idea how words at that distance should relate. Just feeding it more long examples does not help, because the angles themselves are alien. The fix is to slow the hand down so positions up to 32 still land inside the 0-to-4 range it understands. That remapping is RoPE scaling, and a short training phase then teaches the model to settle into the rescaled dial.

Key concepts

Rotary position embeddings Position interpolation YaRN Context-length extension

Quick scan · 2–4 min

Everything important, quickly.

4 min: RoPE position encoding + why angles beyond the trained length are off-distribution + why FT data alone fails + the rescaling family (linear, NTK, YaRN) + the short continued-training phase + cost versus pretraining.

Side-by-side

Method	How it rescales	Trade-off
Linear position interpolation	Divide every position by the scale factor	Simple; blurs high-frequency local resolution
NTK-aware scaling	Change the RoPE frequency base	Keeps high frequencies sharp; can be near zero-shot
YaRN	Per-frequency ramp plus attention-temperature scaling	Best quality per training token; more moving parts
Pure long-context FT only	No rescaling at all	Fails; inputs stay off-distribution

Real-world examples

Real products, models, and research that use this idea.

Meta's Llama 4 long-context variants extend the window via RoPE frequency scaling followed by continued training on long documents.
The YaRN method underpins many open-weight long-context releases, including extended Mistral and Qwen checkpoints on Hugging Face.
Together.ai and Unsloth publish recipes that apply NTK-aware or YaRN RoPE scaling plus a short LoRA fine-tune to push base models to 32k or 128k.
DeepSeek V4 and Qwen long-context models document position-interpolation style RoPE scaling as the basis for their extended windows.
Hugging Face transformers exposes a rope_scaling config (linear, dynamic NTK, yarn) so practitioners enable extension without retraining from scratch.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does linear position interpolation blur local resolution while NTK-aware scaling preserves it?

Sign in or go Pro to see the approach for this follow-up.

QHow many tokens of continued training do you actually need after rescaling, and why so few?

Sign in or go Pro to see the approach for this follow-up.

QWhy does YaRN add an attention-temperature term, and what failure does it address?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.