Defend a mechanistic explanation for why lost-in-the-middle appears in RoPE based transformers

Short answer·Hard·4.0 · 0·~3 min·Asked atBaseten Linkedin Palantir

Attempt it

Several mechanistic theories try to explain lost-in-the-middle. Walk through the strongest two, naming the model components involved.

Free · 2 AI evals / day

**Lost in the middle is the empirical observation that LLMs attend strongly to the start and end of long contexts and skim the middle.** Plot accuracy on a needle in a haystack task against the needle's position and you get a U shaped curve: high at both ends, low in the middle. The question is why. Two mechanistic explanations have the most evidence behind them, and they compound. **Attention sinks.** Xiao et al. (StreamingLLM,2023) showed that in trained transformers a few early tokens, often the beginning of sequence token or its immediate neighbors, absorb a disproportionate share of softmax attention mass across many heads and layers. The softmax has to put its mass somewhere; if no token in the middle is a strong match for a query, the residual mass flows to these sink tokens as a kind of attention overflow. The result is that the leading tokens always have high attention weight, even when their content is uninformative. **RoPE position bias.** Rotary positional encoding rotates query and key vectors by position dependent angles. The dot product `q_m · k_n` ends up depending on `(m - n)`, the relative distance. Empirically the geometric magnitude of that dot product decays with distance, tokens far apart in position contribute less to attention than tokens close together. So recent tokens (small `m - n`) win attention from the current query. **Compounding.** Start wins by sink, end wins by recency, middle is starved on both fronts. That is the U curve.

**The U shaped lost in the middle curve has two compounding mechanistic causes, and most rigorous treatments invoke both.** **Attention sinks (softmax mass dynamics).** Characterized by Xiao et al. in StreamingLLM (2023). In a trained transformer, the softmax over attention scores forces all weight to sum to one. When no token in the recent window is a strong key match for the current query, the residual mass flows somewhere predictable, empirically, to a handful of early tokens, often near the start of sequence position. These tokens become attention sinks: they receive substantial weight regardless of their content. Removing them at inference time tanks model quality even for context that supposedly does not depend on them. The consequence for lost in the middle: the leading tokens enjoy persistent high attention weight by virtue of being sinks. They are favored even when their semantic content is irrelevant. **RoPE relative position decay.** Rotary positional encoding (Su et al. 2021) rotates query and key vectors by position dependent angles `θ_m = m / 10000^{2i/d}`. The resulting dot product `q_m · k_n` becomes a function of the relative position `(m - n)`. The geometric magnitude of this function decays as `|m - n|` grows, particularly under length extrapolation beyond the training context. Recent tokens (the end of the sequence) are close to the current query in position and benefit from this decay shape. Tokens in the middle are farther from the active end of the window and get a smaller geometric weight. **The compound is the U shape.** The start of the context wins attention through the sink mechanism. The end wins through recency (small relative distance). The middle has neither advantage, it is too far from the active query for RoPE proximity and not at a sink position.

TL;DR

The U shaped lost in the middle curve has two compounding mechanistic causes: attention sinks hoard softmax mass at the start, and RoPE relative position decay weakens long range pairs so the middle is doubly starved.

Memory aid

Easy to grasp

Imagine a long lecture hall where students raise hands to ask the teacher questions. By habit, the teacher always glances first at the people sitting closest to the door (sink seats), and naturally pays the most attention to whoever just spoke (recency). The students in the middle rows raised their hands too, but they get squeezed out from both ends, the front gets attention by tradition, the back gets attention by proximity in time, and the middle just sits there waving. Lost in the middle is exactly that compounded squeeze, with the front being attention sinks and the back being recent tokens.

Key concepts

Context engineering

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Lost in the middle is one of the most important empirical findings about long context LLMs, and one of the most frequently invoked at the context engineering layer. Liu et al. (2023) plotted accuracy on a needle in a haystack task against the needle's position in the prompt and produced the canonical U shape: high accuracy at the start and end of the context, low in the middle.

This question asks for the mechanistic explanation, not just the observation. The two strongest candidates, attention sinks and RoPE relative position decay, both have empirical support and compound to produce the curve. A senior or staff level answer names both, ties each to specific model components, and references the architectural mitigations that have emerged.

The empirical observation

The lost in the middle effect describes a position dependent accuracy curve on retrieval and reasoning over long contexts. The classic experiment is needle in a haystack (NIAH): place a single fact (the needle) at varying positions in a long irrelevant context (the haystack), ask a question that requires the needle, and measure accuracy as a function of needle position.

Across many models and many evaluation suites (RULER, BABILong, the original Liu et al. 2023 setup), the resulting curve is approximately U shaped. Accuracy is high when the needle lives in the first 10 to 20 percent of the context, drops sharply into the middle (often by 20 to 40 percent of absolute accuracy), and recovers as the needle approaches the end.

The shape is robust across architectures (with different magnitudes) and across model sizes. It has been observed in Llama, Mistral, Qwen, Claude, GPT, and Gemini families. Mitigations flatten the curve but rarely eliminate it. The question is what mechanism in the transformer produces this specific shape, and whether one mechanism is sufficient or both compound.

Mechanism one: attention sinks

Mechanism two: RoPE relative position decay

Why the two mechanisms compound

Mitigations and their tradeoffs

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

StreamingLLM (Xiao et al. 2023) introduced the attention sink concept and showed that preserving the first few tokens stabilizes long context inference dramatically.
YaRN (Peng et al. 2023) extends RoPE's effective range via frequency rescaling and is implemented in Llama 3.x, Mistral, and Qwen 2.x model families.

What an interviewer would ask next. Try answering before peeking at the approach.

QIf you swap RoPE for ALiBi, do attention sinks still appear?

Yes. Attention sinks are a property of softmax conservation under weak query key matches, not a property of any specific positional encoding. The sink mechanism has been observed across encodings, including ALiBi based models. Switching the positional encoding changes the recency side of the curve (how recency decays) but does not eliminate the start of context advantage from sinks.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Blaming lost in the middle on a single cause (just RoPE, just attention sinks) without naming the compounding interaction between them.

60 second bullets to scan on the way to the call.

What lost in the middle means as an empirical observation
The attention sink mechanism and the softmax conservation argument

Primary sources. Browse if you want the original framing.