1Evaluate on long-context benchmarks (e.g. needle-in-a-haystack, passkey) to confirm long-range retrieval works without regressing short-context quality.
2Briefly fine-tune the model on long-context data so attention weights adapt to the new effective rotation frequencies.
3Optionally swap linear scaling for YaRN, which rescales different RoPE frequency bands non-uniformly to reduce short-context degradation.
4Scale the position indices fed into RoPE at attention time: pos' = pos / scale_factor, where scale_factor = target_length / training_length (here 16k / 4k = 4).
**Position interpolation (PI)** extends a RoPE model's context length without retraining from scratch. Chen et al. 2023 showed a 4k-trained model goes to 16k or 32k with a tiny fine-tune by rescaling positions.
### The recipe in order
1. **Rescale positions** (cheap). Divide the position index by the scale factor: `pos' = pos / 4` for 4k -> 16k. The RoPE angles now span the training-time range across longer context.
2. **Fine-tune at long context** (the work). Run briefly on long-context data. The Q and K projections adapt to the rescaled spectrum. Without this, short-context quality drops significantly.
3. **Evaluate** on long-context benchmarks. Needle-in-a-haystack and passkey tests check long-range retrieval before declaring success.
4. **Optionally swap to YaRN**. If short-context quality regressed too much, replace linear `pos / scale` with YaRN's per-band scaling: stretch slow bands more, fast bands less. Brief re-fine-tune.
### Why this order
Step 1 is cheap and changes the model's view of positions; step 2 lets the model adapt; step 3 confirms it worked; step 4 is the refinement when the basic recipe is not enough.
**Position interpolation** (Chen 2023) is the canonical recipe for extending RoPE models to longer context. Order matters: each step depends on the previous.
### Step 1: Rescale positions at attention
The cheap algebraic trick. Divide the position index by the scale factor before computing RoPE angles:
```python
scale_factor = target_length / training_length # 16k/4k = 4
angles = (positions / scale_factor) / base ** (2 * i / d_head)
```
Maps long-context positions back into the training-time phase range. No training; just precompute the new tables.
### Step 2: Fine-tune at target context
Mandatory adaptation. Run the model on long-context data (typically 100M to a few billion tokens at target length). Q, K, FFN, and output projections adapt to the rescaled spectrum. Without this, short-context quality drops 30-50%.
### Step 3: Evaluate
Needle-in-a-haystack, passkey, RULER, LongBench, plus short-context regression checks. A model that 'extends to 16k' but fails needle-in-a-haystack at 12k is not actually functional.
### Step 4 (optional): Swap to YaRN
Uniform PI slows ALL frequency bands, hurting short context. YaRN scales bands non-uniformly: aggressive on slow bands, mild on fast bands. Better quality across short and long context. Used in 1M-token models like Qwen 3.5. Requires another brief fine-tune.
### Why the order is fixed
Cannot evaluate before rescaling and fine-tuning (model is not long-context yet). Cannot fine-tune before rescaling (nothing to adapt to). YaRN is a refinement of PI's baseline, not an alternative pipeline.
Quick reference
TL;DR
Rescale positions first (cheap, no training), then fine-tune at long context (mandatory adaptation), then evaluate, then optionally swap linear scaling for YaRN for better short-context quality.
Picture stretching a song to play in slow motion so it lasts four times longer. First you change the playback speed (cheap, instant). The instruments now sound weird because they were not designed for that tempo, so you give the band a brief practice session to adapt their playing (the fine-tune). Then you record a test concert to check it sounds good. Finally, if some instruments still sound off, you could swap the uniform slowdown for a smarter scheme that slows down only certain pitches, that is YaRN.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Models that do not use RoPE (ALiBi, T5 bucketed bias, learned PE) PI specifically rescales RoPE rotation angles. It does not apply to other positional encoding schemes. For ALiBi, the equivalent extension is just running it at longer context (it length-extrapolates natively). For learned PE, there is no clean extension recipe; you usually have to retrain.
Extreme context extensions (more than 16x training length) Uniform PI's short-context regression grows with the scale factor. Beyond about 16x extension (4k -> 64k), YaRN or LongRoPE is essentially mandatory for acceptable short-context quality. Pure PI is acceptable up to about 8x.
When fine-tuning budget is zero PI without fine-tuning has severe short-context regression (perplexity can jump 30-50%). If you cannot afford even a brief fine-tune, you cannot afford to extend context properly. Consider sliding-window attention or RAG instead.
Native long-context pretraining is feasible If you can afford to pretrain from scratch at long context (e.g., training a new model with high RoPE base and 128k native context throughout pretraining), you skip the extension recipe entirely. This is what 2026 frontier models do, but it is much more expensive than extending an existing model via PI.
Quick scan · 2–4 min
Everything important, quickly.
Order the four steps (rescale, fine-tune, evaluate, optionally YaRN), explain each step's role, justify the order dependencies, and mention YaRN as the refinement for the short-context cost of uniform scaling.
Real-world examples
Real products, models, and research that use this idea.
1Llama 4 long-context variants use a PI-like recipe combined with YaRN for native 128k-1M context support.
2Qwen 3.5's 1M-token context was achieved via base raising plus YaRN-style per-band scaling plus continued pretraining.
3Vicuna and early Llama-2 community extensions used pure PI (Chen 2023 recipe) for 4k -> 16k and 4k -> 32k extensions.
4Mistral's 32k context support combined sliding-window attention with PI-style extension of the underlying RoPE.
5DeepSeek V4's long-context recipe combines high-base RoPE pretraining with YaRN-style refinement during extension.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat benchmark would you use to distinguish a model with 'real' 128k context from one with '128k context that aliases past 32k'?
Sign in or go Pro to see the approach for this follow-up.
QWhy does uniform PI scaling cost short-context quality, and how does YaRN avoid it?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Fine-tuning BEFORE rescaling positions. The rescale is the cheap algebraic prerequisite; the fine-tune is what adapts the model to the rescaled spectrum.