Order the pipeline that stretches an 8k pretrained model to 128k context

Order steps·Hard·4.0 · 0·~1 min·Asked atHaptik Perplexity Swiggy

Attempt it

1Pick the target context length and derive the new RoPE base theta or frequency scaling factor
2Gate the release on long context evals such as needle in a haystack and RULER, not just short context perplexity
3Continue pretraining on a mixture rich in genuinely long sequences at the new length
4Apply the rescaled rotary frequencies to the pretrained checkpoint
5Run long context instruction tuning so chat behavior survives at extended lengths

Context extension is a five step pipeline. Each step exists for a reason and skipping one shows up at the next step. **Step 1.** Pick the new target length and derive a RoPE scaling. The original model was trained with RoPE at one base frequency suited to 8k. At 128k, the same frequencies produce rotation angles between distant tokens that the model never saw during training. You rescale: raise the base theta or apply NTK-aware or YaRN scaling. **Step 2.** Apply the rescaled frequencies to the existing checkpoint. The model now uses slower rotations, so far-apart tokens look like nearer ones did during pretraining. **Step 3.** Continue pretraining on a long-data mixture. Without this step, the model has the right positional geometry but has never seen reasoning over 100k tokens. Quality stays mediocre. **Step 4.** Redo instruction tuning at the new length. Chat behavior was tuned at 8k; it does not automatically survive the extension. **Step 5.** Gate on long-context evals: RULER, needle in a haystack, LongBench. Short-context perplexity stays clean even when long-context reasoning is broken.

Five stages, in order: target selection, architectural rescaling, long-context continued pretraining, long-context instruction tuning, long-context evaluation. **Why architecture first.** RoPE rotates Q and K by position-dependent angles. The base frequencies are chosen at pretraining time for a specific length. Extending the model without rescaling produces rotation angles at long distances that the model never saw, and quality falls off a cliff. Common scalings include linear position interpolation, NTK-aware scaling, and YaRN. Llama 3 uses base theta 500k (versus Llama 2's 10k) to natively reach 128k. **Why continued pretraining matters.** The model has been told there is a longer board, but it has not played any games on it. Continued pretraining on a data mixture rich in genuinely long sequences (full books, long technical documents, long code repositories) teaches the model to actually use the extended range. **Why instruction tuning is redone.** The first SFT pass was on short conversations. Long-context chat needs its own tuning, otherwise the model performs well on benchmarks and badly on real users with long sessions. **Why evals gate the release.** Perplexity at long context stays low even when retrieval fails. Needle in a haystack, RULER, and LongBench reveal whether the model can actually use the context. Skipping this step ships a model with a long context number on the spec sheet and a short context behavior in production. $$\theta_m = m / \text{base}^{2i/d}$$

TL;DR

Pick the target length, derive new RoPE frequencies, apply them, continue pretraining on long sequences, redo long-context instruction tuning, then gate on long-context evals before shipping.

Memory aid

Easy to grasp

Imagine teaching a child to read full novels when they have only ever read picture books. You cannot just hand them a 500 page book and hope. First, you change the way the page numbers are written so the child can still tell page 387 apart from page 388 in a long book; the existing numbering scheme would just blur them together. Then you give the child many real long books to practice on, not the same picture book stretched out. Next you let them practice answering questions about long stories so their conversation skills survive. Only after all that do you test: can they actually find a specific detail buried in chapter 14? If yes, ship.

Key concepts

Transformer Architecture

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Extending a pretrained model from 8k context to 128k or 1M is one of the highest leverage moves an LLM team can make, and one of the easiest to get wrong. The pipeline has five stages: pick the target and rescale positions, apply the rescaling to the checkpoint, continue pretraining on long-data mixtures, redo instruction tuning at the new length, and gate the release on long-context evaluations.

The order is not interchangeable. Each stage provides a precondition the next stage assumes. Skip continued pretraining and the model never learns to use the extended range, no matter how good the positional surgery was. Skip long-context evals and you ship a model that retrieves from short windows and hallucinates everywhere else.

This deep dive walks through each stage with the reasoning, the failure modes, and the actual recipes shipped by Llama 3, DeepSeek-V3, and similar 2026 production models.

Stage 1 and 2: pick the target and rescale RoPE

RoPE encodes position by rotation. Each pair of dimensions in Q and K gets rotated by an angle that depends on position and a per-dimension frequency. The frequencies are derived from a base theta, typically 10000 in the original RoPE paper.

\theta_m = m / \text{base}^{2i/d}

At the pretraining length, the model sees rotation angles between Q and K that span a known range. Extend the sequence to 128k with the same base theta and the rotation angles at long distances are completely outside that range. The attention dot product depends on cos(theta_m - theta_n), which aliases or saturates at long m and n the model never saw. Quality on long-range retrieval collapses.

Rescaling fixes the geometry. Three common approaches: linear position interpolation (PI) compresses positions before applying RoPE; NTK-aware scaling raises the base theta so high frequencies remain intact; YaRN combines NTK with per-dimension correction terms. Llama 3 chose base theta 500k at pretraining time so 128k is reachable without further rescaling; Llama 3.1 extends Llama 3 models further with similar staging.

Apply to the checkpoint. With the new frequencies in place, every layer's attention now produces in-distribution rotation angles even at 100k tokens. This step is mechanical, but if you skip the rescaling or apply it wrong, the next stage runs at wrong geometry and the compute is mostly wasted.

Stage 3: continued pretraining on long sequences

Stage 4: long-context instruction tuning

Stage 5: long-context evaluation as the release gate

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Llama 3.1 extended 8B and 70B models from 8k to 128k using rescaled RoPE plus continued pretraining on long data, then redone instruction tuning, gated on RULER scores
DeepSeek-V3 ships at 128k context via YaRN-style RoPE scaling and a long-context continued pretraining phase, validated on long-context benchmarks before release

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you split the continued pretraining budget between long-only and mixed data?

A pure long-only mixture degrades short-context quality. Most recipes use 70-90% mixed and 10-30% deliberately long, with the long fraction front-loaded to teach the new range and then taper toward mixed to preserve base abilities.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Rescaling RoPE theta and shipping, skipping the long-context continued pretraining that actually teaches the model to use the extended range.

60 second bullets to scan on the way to the call.

Why RoPE base theta or scaling must change before continued pretraining
Why continued pretraining on long data is non-optional

Primary sources. Browse if you want the original framing.