Long-context language models do not attend evenly across the full prompt. Empirical studies and production logs show a U-shape: the start of the context and the end of the context get attended strongly, and the middle gets attended weakly. This is the **lost-in-the-middle** effect.
The practical question is what to do about it. The mitigations that actually work exploit the shape of the curve rather than fight it. There are three.
First, **place the most relevant retrieved chunks at the very top and bottom** of the retrieved block. The top gets the primacy attention boost; the bottom gets the recency boost. Sandwich the less-critical material in the middle where attention is weaker anyway.
Second, **rerank and trim** to keep context dense. A cross-encoder rerank lets you keep the top three to five chunks and drop the rest. The U-curve gets less punishing when the context is short, because there is less middle to be lost in.
Third, **restate the user question right before the answer position**. The model is about to generate, and the recency end is where it pays most attention. Anchoring the question there keeps the model on task.
The failed mitigations come from a "more is better" reflex. Stuffing more chunks deepens the trough. Adding a bare "pay attention to this" marker is unreliable, the model does not magically focus more on a sentence that asks for focus. Switching to a bigger advertised context window without changing retrieval just spreads the U-curve over a longer span; the trough still exists.
**Lost-in-the-middle** is the empirical observation that long-context models attend strongly to the start and end of the prompt and weakly to the middle. The robust mitigations exploit that shape; the failed ones fight it with verbal nudges or more tokens.
**Head and tail placement.** Sort retrieved chunks by relevance, then interleave: highest-rerank goes at the top of the retrieval block, second-highest at the bottom, and so on. Lower-relevance material lands in the middle where attention is already weak. This is essentially free, it costs one sort, and it reliably lifts answer accuracy on multi-document QA.
**Rerank and trim.** A cross-encoder rerank over a wide candidate pool (top-50) followed by aggressive trimming to top-3 to top-5 keeps the context dense. A short context has less middle, so the U-curve has less surface area to hurt you. Cohere Rerank 3, BGE rerankers, and Voyage Rerank-2 are the typical 2026 choices.
**Question restatement near the answer.** Place the user's question at the recency end, right before the model generates. The model attends most to the bottom of the prompt and is about to write from there; reminding it of the question there keeps the answer aligned.
**Why the other options fail.** Adding more chunks raises the chance the right one is in there, but it widens the trough and dilutes attention. Markers like "pay attention to this" are unreliable, structural fixes beat textual pleas. Switching to a larger advertised context window without changing retrieval inherits the same U-curve across a longer span; RULER and BABILong evaluations confirm that nominal capacity is not effective capacity.
The pattern across the working mitigations: change where the model looks, not what you ask it to look at.
Quick reference
TL;DR
Move the most relevant content to the start or end of the retrieved block, rerank and trim to keep context dense, and restate the question near the answer position.
Imagine reading a thirty-page printout to answer a single question. You always look at the first and last pages most carefully; you skim the middle. Even if someone hands you sixty pages instead of thirty, your eyes still favor the edges. The trick is not to give you more paper, it is to put the load-bearing facts on page one and the last page, and to repeat the question right before the answer goes. A sticky note saying "pay attention" on page fifteen does very little. The same is true for a language model reading a long context: structural placement beats verbal nudges every time.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Short prompts where retrieval fits in 2-3 chunks and the total context is under 4K tokens The U-curve only emerges at scale; below the threshold the layout choice does not matter
Single-document QA where there is no choice over chunk ordering The mitigations are about retrieval composition; with one document the only option is to summarize
Streaming or one-shot pipelines where you cannot rerank synchronously Rerank and trim adds a forward pass of latency; if the budget is tight, accept the U-curve and instead use a smaller chunk set
Quick scan · 2–4 min
Everything important, quickly.
6 minutes: the U-curve, three structural mitigations that work, three reflexes that fail, and how to measure curve shape on a new model.
Code example
python
def head_and_tail_layout(chunks_sorted_desc: list) -> list:
"""
Interleave so highest-rerank chunks land at position 0 and -1,
next-highest at position 1 and -2, and so on. Lowest scores in middle.
"""
head, tail = [], []
for i, chunk in enumerate(chunks_sorted_desc):
(head if i % 2 == 0 else tail).append(chunk)
return head + list(reversed(tail))
def build_prompt(user_q: str, chunks_sorted_desc: list, system: str) -> str:
laid_out = head_and_tail_layout(chunks_sorted_desc)
retrieval_block = '\n\n'.join(f'[{i+1}] {c.text}' for i, c in enumerate(laid_out))
return (
f'{system}\n\n'
f'User question (for context): {user_q}\n\n'
f'Retrieved evidence:\n{retrieval_block}\n\n'
f'User question (answer this now): {user_q}\n'
)
Real-world examples
Real products, models, and research that use this idea.
1Cohere Rerank 3 and BGE reranker-v2 are the 2026 production defaults for the rerank and trim leg; both expose a cross-encoder scoring path used to shrink top-50 candidate pools to top-5.
2Anthropic and OpenAI long-context evaluations both publish position-conditioned accuracy curves; the U-shape persists across 200K-2M nominal windows.
3LangChain's `LongContextReorder` post-processor implements the head and tail layout out of the box; LlamaIndex offers an equivalent `LongContextReorder` node.
4Cursor's repo-grounded coding agent uses rerank-trim plus question-restatement to keep the active file content at the recency end before the model edits.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you measure the shape of the curve on a new production model?
Sign in or go Pro to see the approach for this follow-up.
QIf head and tail layout makes accuracy worse on one of your eval slices, what does that tell you?
Sign in or go Pro to see the approach for this follow-up.
QHow does prompt caching interact with head and tail layout when retrieval changes per turn?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reaching for a bigger model context window or stuffing more chunks instead of restructuring retrieval. Both make the U-curve worse, not better.