Defend a mechanistic explanation for why lost-in-the-middle appears in RoPE based transformers

Short answer·Hard·4.0 · 0·~3 min·Asked atBaseten Linkedin Palantir

Attempt it

Several mechanistic theories try to explain lost-in-the-middle. Show the strongest two, naming the model components involved.

Free · 2 AI evals / day

**Lost in the middle is the empirical observation that LLMs (large language models) use information at the start and end of long contexts best and skim the middle.** Liu et al. (2023) moved the relevant document through a long prompt and got a U shaped accuracy curve: high at both ends, low in the middle. Two mechanisms explain it, and they compound. **Attention sinks.** Xiao et al. (StreamingLLM, 2023) showed that a few early tokens absorb a large share of attention mass in trained transformers. The softmax has to put its mass somewhere. If no token is a strong match for a query, the leftover mass flows to these sink tokens as attention overflow. So the leading tokens always get high attention weight, even when uninformative. **RoPE position bias.** Rotary positional encoding rotates query and key vectors by position dependent angles. The dot product `q_m · k_n` ends up depending on `(m - n)`, the relative distance, and it tends to shrink as that distance grows. So recent tokens (small `m - n`) win attention from the current query. **Compounding.** Start wins by sink, end wins by recency, middle is starved on both fronts. That is the U curve.

**The U shaped lost in the middle curve has two compounding mechanistic causes.** **Attention sinks (softmax mass dynamics).** Xiao et al. characterized them in StreamingLLM (2023). The softmax over attention scores forces all weight to sum to one. When no token is a strong key match for the current query, the leftover mass goes to a handful of early tokens. These tokens become attention sinks: they receive substantial weight regardless of their content. Removing them at inference time badly hurts model quality even for context that supposedly does not depend on them. **RoPE relative position decay.** Rotary positional encoding (Su et al. 2021) rotates query and key vectors by position dependent angles `θ_m = m / 10000^{2i/d}`. The resulting dot product `q_m · k_n` becomes a function of the relative position `(m - n)`. Its magnitude tends to decay as `|m - n|` grows. Recent tokens sit close to the current query and benefit. Middle tokens are farther away and get a smaller weight. **The compound is the U shape.** The start of the context wins attention through the sink mechanism. The end wins through recency (small relative distance). The middle has neither advantage: it is too far from the active query for RoPE proximity and not at a sink position. Mitigations target each side. StreamingLLM keeps sink tokens in the cache, and Position Interpolation and YaRN rescale RoPE to soften the decay. Neither removes the curve.

TL;DR

The U shaped lost in the middle curve has two compounding mechanistic causes: attention sinks hoard softmax mass at the start, and RoPE relative position decay weakens long range pairs so the middle is doubly starved.

Memory aid

Easy to grasp

Imagine a long lecture hall where students raise hands to ask the teacher questions. By habit, the teacher always glances first at the people sitting closest to the door (sink seats). The teacher naturally pays the most attention to whoever just spoke (recency). The students in the middle rows raised their hands too, but they get squeezed out from both ends. The front gets attention by tradition, the back gets attention by proximity in time, and the middle just sits there waving. Lost in the middle is exactly that compounded squeeze, with the front being attention sinks and the back being recent tokens.

Key concepts

Context engineering

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Lost in the middle is one of the most important empirical findings about long-context LLMs. It is also one of the most frequently invoked at the context engineering layer. Liu et al. (2023) plotted accuracy on multi-document question answering and key-value retrieval against the position of the relevant information in the prompt. The result was the well-known U shape: high accuracy at the start and end of the context, low in the middle.

The two strongest mechanistic explanations, attention sinks and RoPE relative position decay, both have empirical support and compound to produce the curve.

The empirical observation

The lost in the middle effect describes a position dependent accuracy curve on retrieval and reasoning over long contexts. The original experiment (Liu et al. 2023) used multi-document question answering. Place the one document that holds the answer at varying positions in a set of 10, 20 or 30 retrieved documents. Then ask the question and measure accuracy as a function of that document's position. A synthetic key-value retrieval task showed the same pattern.

The resulting curve is U shaped. Accuracy is highest when the relevant document is at the start or the end of the context and lowest in the middle. For GPT-3.5-Turbo, accuracy dropped by more than 20 points from the best position to the worst. In the 20- and 30-document settings, its worst case fell below its score with no documents at all.

Liu et al. found the shape in open models (MPT-30B-Instruct, LongChat-13B) and closed ones (GPT-3.5-Turbo, Claude-1.3), including models built for long context. Mitigations flatten the curve but rarely eliminate it. The question is what mechanism in the transformer produces this specific shape, and whether one mechanism is sufficient or both compound.

Mechanism one: attention sinks

Mechanism two: RoPE relative position decay

Why the two mechanisms compound

Mitigations and their trade-offs

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

StreamingLLM (Xiao et al. 2023) introduced the attention sink concept and showed that preserving the first few tokens greatly stabilizes long context inference.
YaRN (Peng et al. 2023) extends RoPE's effective range via frequency rescaling plus an attention temperature. Qwen2.5 models enable it with a yarn rope_scaling entry for long inputs, while Llama 3.1 uses its own frequency-dependent scaling (rope_type: llama3).

What an interviewer would ask next. Try answering before peeking at the approach.

QIf you swap RoPE for ALiBi, do attention sinks still appear?

Yes. Attention sinks are a property of softmax conservation under weak query key matches, not a property of any specific positional encoding. The sink mechanism has been observed across encodings, including ALiBi based models. Switching the positional encoding changes the recency side of the curve (how recency decays) but does not eliminate the start of context advantage from sinks.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Blaming lost in the middle on a single cause (just RoPE, just attention sinks) without naming the compounding interaction between them.

60 second bullets to scan on the way to the call.

What lost in the middle means as an empirical observation
The attention sink mechanism and the softmax conservation argument

Primary sources. Browse if you want the original framing.