Order the three steps of a position-interpolation fine-tune that takes RoPE from 4k to 16k.

Order steps·Medium·4.0 · 0·~1 min·Relevant atMeta

Attempt it

1Evaluate on long-context benchmarks (e.g. needle-in-a-haystack, passkey) to confirm long-range retrieval works without regressing short-context quality.
2Briefly fine-tune the model on long-context data so attention weights adapt to the new effective rotation frequencies.
3Optionally swap linear scaling for YaRN, which rescales different RoPE frequency bands non-uniformly to reduce short-context degradation.
4Scale the position indices fed into RoPE at attention time: pos' = pos / scale_factor, where scale_factor = target_length / training_length (here 16k / 4k = 4).

**Position interpolation (PI)** extends a RoPE model's context length without retraining from scratch. Chen et al. 2023 showed a 4k-trained model goes to 16k or 32k with a tiny fine-tune by rescaling positions. ### The recipe in order 1. **Rescale positions** (cheap). Divide the position index by the scale factor: `pos' = pos / 4` for 4k -> 16k. The RoPE angles now span the training-time range across longer context. 2. **Fine-tune at long context** (the work). Run briefly on long-context data. The Q and K projections adapt to the rescaled spectrum. Without this, short-context quality drops significantly. 3. **Evaluate** on long-context benchmarks. Needle-in-a-haystack and passkey tests check long-range retrieval before declaring success. 4. **Optionally swap to YaRN**. If short-context quality regressed too much, replace linear `pos / scale` with YaRN's per-band scaling: stretch slow bands more, fast bands less. Brief re-fine-tune. ### Why this order Step 1 is cheap and changes the model's view of positions; step 2 lets the model adapt; step 3 confirms it worked; step 4 is the refinement when the basic recipe is not enough.

**Position interpolation** (Chen 2023) is the canonical recipe for extending RoPE models to longer context. Order matters: each step depends on the previous. ### Step 1: Rescale positions at attention The cheap algebraic trick. Divide the position index by the scale factor before computing RoPE angles: ```python scale_factor = target_length / training_length # 16k/4k = 4 angles = (positions / scale_factor) / base ** (2 * i / d_head) ``` Maps long-context positions back into the training-time phase range. No training; just precompute the new tables. ### Step 2: Fine-tune at target context Mandatory adaptation. Run the model on long-context data (typically 100M to a few billion tokens at target length). Q, K, FFN, and output projections adapt to the rescaled spectrum. Without this, short-context quality drops 30-50%. ### Step 3: Evaluate Needle-in-a-haystack, passkey, RULER, LongBench, plus short-context regression checks. A model that 'extends to 16k' but fails needle-in-a-haystack at 12k is not actually functional. ### Step 4 (optional): Swap to YaRN Uniform PI slows ALL frequency bands, hurting short context. YaRN scales bands non-uniformly: aggressive on slow bands, mild on fast bands. Better quality across short and long context. Used in 1M-token models like Qwen 3.5. Requires another brief fine-tune. ### Why the order is fixed Cannot evaluate before rescaling and fine-tuning (model is not long-context yet). Cannot fine-tune before rescaling (nothing to adapt to). YaRN is a refinement of PI's baseline, not an alternative pipeline.

Quick reference

TL;DR

Rescale positions first (cheap, no training), then fine-tune at long context (mandatory adaptation), then evaluate, then optionally swap linear scaling for YaRN for better short-context quality.

Memory aid

Easy to grasp

Picture stretching a song to play in slow motion so it lasts four times longer. First you change the playback speed (cheap, instant). The instruments now sound weird because they were not designed for that tempo, so you give the band a brief practice session to adapt their playing (the fine-tune). Then you record a test concert to check it sounds good. Finally, if some instruments still sound off, you could swap the uniform slowdown for a smarter scheme that slows down only certain pitches, that is YaRN.

Key concepts

Position interpolation YaRN RoPE

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Order the four steps (rescale, fine-tune, evaluate, optionally YaRN), explain each step's role, justify the order dependencies, and mention YaRN as the refinement for the short-context cost of uniform scaling.

Real-world examples

Real products, models, and research that use this idea.

Llama 4 long-context variants use a PI-like recipe combined with YaRN for native 128k-1M context support.
Qwen 3.5's 1M-token context was achieved via base raising plus YaRN-style per-band scaling plus continued pretraining.
Vicuna and early Llama-2 community extensions used pure PI (Chen 2023 recipe) for 4k -> 16k and 4k -> 32k extensions.
Mistral's 32k context support combined sliding-window attention with PI-style extension of the underlying RoPE.
DeepSeek V4's long-context recipe combines high-base RoPE pretraining with YaRN-style refinement during extension.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat benchmark would you use to distinguish a model with 'real' 128k context from one with '128k context that aliases past 32k'?

QWhy does uniform PI scaling cost short-context quality, and how does YaRN avoid it?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Fine-tuning BEFORE rescaling positions. The rescale is the cheap algebraic prerequisite; the fine-tune is what adapts the model to the rescaled spectrum.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.