Extending a 4k-context model to 32k via FT: how RoPE scaling fits in

Attempt it

You want to extend a base model trained on 4k context to support 32k context via fine-tuning. Why is RoPE the relevant lever, what scaling variants (linear, NTK-aware, YaRN) are commonly used, and why is a short FT pass also needed?

Free · 2 AI evals / day

Quick reference

TL;DR

RoPE encodes position by rotating Q and K. Scaling (linear, NTK-aware, YaRN) remaps frequencies to long positions; a short FT pass lets attention adapt.

Memory aid

Easy to grasp

Imagine a clock whose hands tell the model where each word sits. The model only ever practiced reading the clock for the first 4,000 seconds. Past that, the hands spin into positions it has never seen, so it gets confused. RoPE scaling slows the hands down so the 32,000th word lands at a clock angle the model already recognises. But slowing the hands changes every reading slightly, so the model still needs a little refresher practice on long documents. That short practice is the fine-tuning pass. Slow the clock first, then practice reading it. Skip either step and the model still gets lost on long inputs.

Key concepts

Rotary Position Embedding Position Interpolation YaRN Context-length extension

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: how RoPE encodes position, why unseen angles collapse attention, the three scaling variants, why scaling alone fails on big jumps, and why the light fine-tune is required.

Side-by-side

Variant	What it rescales	Strength	Weakness
Linear (Position Interpolation)	Position indices divided by factor	Simple, stable, easy to apply	Blurs high-frequency local detail
NTK-aware	RoPE base theta, per frequency	Preserves short-range attention	Tuning the base needs care
YaRN	Per-dimension interp plus logit temperature	SOTA quality at 8 to 32 times	More moving parts, bundles an FT pass

Real-world examples

Real products, models, and research that use this idea.

Llama 4 long-context variants reach very large windows using RoPE rescaling combined with a continued-pretraining pass on long documents.
YaRN is implemented in vLLM and Hugging Face transformers as a rope_scaling config, used to extend open-weight bases past their native window.
DeepSeek V4 and Qwen long-context releases document NTK-aware or YaRN style frequency scaling plus long-sequence fine-tuning in their technical reports.
Code-specialised Llama 4 variants extend context by adjusting the RoPE base frequency before fine-tuning on long code files.
Together.ai and Modal Labs publish recipes that set a YaRN rope_scaling factor and run a short long-context fine-tune on packed documents.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does NTK-aware scaling preserve short-range attention better than linear Position Interpolation?

QRoughly how much data and which documents would you pick for the long-context fine-tune?

QHow would you evaluate that the extended model truly uses the long context, not just survives it?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Thinking fine-tuning alone extends context. Without rescaling RoPE, the model cannot even represent positions past its training range, so long-sequence training stalls.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.