Why does pure long-context FT (without RoPE scaling) fail to extrapolate?

**RoPE** (rotary position embeddings) tells the model where each token sits by rotating its query and key vectors by an angle that depends on the position number. Position 5 gets a small rotation, position 4000 a large one. The catch: during pretraining at, say, 4k context, the model only ever saw rotation angles in that range. It learned what those specific angles mean for how tokens relate to each other. Now you want 32k context. Position 32000 produces a rotation angle the model has never encountered. To the model, the input looks alien. It has no learned attention pattern for that angle, so the output degrades sharply. This is why just fine-tuning on long documents does not work. The problem is not that the model needs more practice. The problem is that the inputs themselves are off-distribution. The angles are outside everything it learned. The fix is RoPE scaling: remap the new, large positions back into the trained angle range (using linear interpolation, NTK-aware scaling, or YaRN), then run a short continued-training phase so the model settles into the rescaled positions.

**RoPE** encodes position by rotating query and key vectors at frequencies tied to the absolute position index. At pretraining time with a 4k window, the model only observes rotation angles within that range. It learns attention patterns for those angles only. Extrapolating to 32k creates positions whose rotation angles fall far outside the trained range. The attention inputs are off-distribution. The model has no learned response to those angles, so quality collapses. This is a property of the inputs, not a lack of training. Throwing more long-context data at a pure fine-tune cannot fix an off-distribution input. The fix is to rescale the rotation frequencies so target positions map back into the trained range. Linear position interpolation divides every position by the scale factor. NTK-aware scaling adjusts the frequency base to spread the squeeze across dimensions. YaRN refines this per frequency and adds attention-temperature scaling. After rescaling, a short continued-training phase on long sequences lets the model adapt to the new positional layout. This costs a tiny fraction of pretraining. That combination, frequency rescaling plus light fine-tuning, is the standard recipe for cheap context extension.