Several mechanistic theories try to explain lost-in-the-middle. Walk through the strongest two, naming the model components involved.
Free · 2 AI evals / day
**Lost in the middle is the empirical observation that LLMs attend strongly to the start and end of long contexts and skim the middle.** Plot accuracy on a needle in a haystack task against the needle's position and you get a U shaped curve: high at both ends, low in the middle. The question is why.
Two mechanistic explanations have the most evidence behind them, and they compound.
**Attention sinks.** Xiao et al. (StreamingLLM,2023) showed that in trained transformers a few early tokens, often the beginning of sequence token or its immediate neighbors, absorb a disproportionate share of softmax attention mass across many heads and layers. The softmax has to put its mass somewhere; if no token in the middle is a strong match for a query, the residual mass flows to these sink tokens as a kind of attention overflow. The result is that the leading tokens always have high attention weight, even when their content is uninformative.
**RoPE position bias.** Rotary positional encoding rotates query and key vectors by position dependent angles. The dot product `q_m · k_n` ends up depending on `(m - n)`, the relative distance. Empirically the geometric magnitude of that dot product decays with distance, tokens far apart in position contribute less to attention than tokens close together. So recent tokens (small `m - n`) win attention from the current query.
**Compounding.** Start wins by sink, end wins by recency, middle is starved on both fronts. That is the U curve.
**The U shaped lost in the middle curve has two compounding mechanistic causes, and most rigorous treatments invoke both.**
**Attention sinks (softmax mass dynamics).** Characterized by Xiao et al. in StreamingLLM (2023). In a trained transformer, the softmax over attention scores forces all weight to sum to one. When no token in the recent window is a strong key match for the current query, the residual mass flows somewhere predictable, empirically, to a handful of early tokens, often near the start of sequence position. These tokens become attention sinks: they receive substantial weight regardless of their content. Removing them at inference time tanks model quality even for context that supposedly does not depend on them.
The consequence for lost in the middle: the leading tokens enjoy persistent high attention weight by virtue of being sinks. They are favored even when their semantic content is irrelevant.
**RoPE relative position decay.** Rotary positional encoding (Su et al. 2021) rotates query and key vectors by position dependent angles `θ_m = m / 10000^{2i/d}`. The resulting dot product `q_m · k_n` becomes a function of the relative position `(m - n)`. The geometric magnitude of this function decays as `|m - n|` grows, particularly under length extrapolation beyond the training context. Recent tokens (the end of the sequence) are close to the current query in position and benefit from this decay shape. Tokens in the middle are farther from the active end of the window and get a smaller geometric weight.
**The compound is the U shape.** The start of the context wins attention through the sink mechanism. The end wins through recency (small relative distance). The middle has neither advantage, it is too far from the active query for RoPE proximity and not at a sink position.
Quick reference
TL;DR
The U shaped lost in the middle curve has two compounding mechanistic causes: attention sinks hoard softmax mass at the start, and RoPE relative position decay weakens long range pairs so the middle is doubly starved.
Imagine a long lecture hall where students raise hands to ask the teacher questions. By habit, the teacher always glances first at the people sitting closest to the door (sink seats), and naturally pays the most attention to whoever just spoke (recency). The students in the middle rows raised their hands too, but they get squeezed out from both ends, the front gets attention by tradition, the back gets attention by proximity in time, and the middle just sits there waving. Lost in the middle is exactly that compounded squeeze, with the front being attention sinks and the back being recent tokens.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Short context tasks well within the training window The U curve flattens drastically at modest context lengths. Lost in the middle is primarily a long context phenomenon; for short prompts the position bias is small enough to ignore.
Tasks where positional reordering is not safe, chronological logs, code with strict dependency order, narrative summarization Reordering chunks to exploit the U curve is the standard context engineering fix, but it assumes chunks are positionally interchangeable. When order carries meaning, you cannot reorder away the position bias and must instead summarize or chunk differently.
Models with non RoPE architectures (ALiBi based, learned absolute PE) The RoPE specific mechanism does not apply. Position bias may still exist via the sink mechanism, but the analytical lever is different. ALiBi has its own length extrapolation properties via the per head linear bias.
Quick scan · 2–4 min
Everything important, quickly.
Two compounding mechanisms produce the U shape. First, attention sinks: softmax conservation forces residual mass onto a small set of early tokens, giving the start of context persistent high weight regardless of content (StreamingLLM, Xiao et al. 2023). Second, RoPE relative position decay: the rotated dot product `q_m · k_n` is a function of `(m - n)` with geometric falloff over distance, favoring recent tokens. Start wins by sink, end wins by recency, middle is doubly starved. Mitigations: StreamingLLM (preserve sinks), YaRN and Positional Interpolation (extend RoPE), positional dropout. The context engineering response is to reorder retrieved chunks so high relevance lands at the ends.
Real-world examples
Real products, models, and research that use this idea.
1StreamingLLM (Xiao et al. 2023) introduced the attention sink concept and showed that preserving the first few tokens stabilizes long context inference dramatically.
2YaRN (Peng et al. 2023) extends RoPE's effective range via frequency rescaling and is implemented in Llama 3.x, Mistral, and Qwen 2.x model families.
3RULER and BABILong benchmark suites in 2026 explicitly probe the U shape across positions, exposing how much of the curve survives each mitigation.
4Claude Opus 4.7 and Gemini 2.x Pro both report long context behaviour that is measurably flatter than 2023 era models, attributed to a combination of attention sink aware training and RoPE extension techniques.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf you swap RoPE for ALiBi, do attention sinks still appear?
Sign in or go Pro to see the approach for this follow-up.
QHow would you empirically test whether the depth of the U is dominated by sinks or by RoPE decay in a specific model?
Sign in or go Pro to see the approach for this follow-up.
QWhy does scaling up the context window in pretraining alone not fully fix the curve?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Blaming lost in the middle on a single cause (just RoPE, just attention sinks) without naming the compounding interaction between them.