Restating the question right before the answer position anchors the task in the recency slot, where attention is strongest, and prevents the model from drifting into 'summarize the evidence' mode.
Imagine asking a friend a specific question, then handing them a 200-page report and asking them to answer using the report. By the time they have read the report, the specific question has faded; they will tend to summarize the report instead of answering you. Now imagine you slip the question onto a sticky note and attach it to the last page. Right when they finish reading and pick up the pen, the question is the freshest thing they saw. They answer the question, not the report. The restated question is that sticky note. It works for the same reason: it puts the goal where attention is strongest, right before the answer.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The question restate is one of the most reliable long-context tricks in production RAG. It costs roughly 30 tokens, takes a one-line change to your prompt template, and consistently improves accuracy on long-context QA tasks by 5 to 15 percent. The mechanism is recency exploitation: by placing the question right before the answer position, you put the task in the strongest-attention slot, where the model's autoregressive next-token prediction is most steered.
The failure it prevents is on topic but off question. The user asks a specific question; the retrieval pipeline returns relevant chunks; the model produces an answer that is grounded in the chunks but never actually addresses the question. This deep dive covers why the failure happens, how the restate fixes it, and the production variants that work best.
Mental model: the question at the top is upstream evidence. The question at the bottom is downstream task. Put the task in the strongest-attention slot.
The on topic but off question failure mode
The shape of the failure
A RAG pipeline is set up with: system prompt at the top, user question, then 5,000 to 20,000 tokens of retrieved chunks, then the model generates the answer. The model has all the evidence and the question is technically in the prompt. The answer it produces is well-written, factually grounded in the retrieved content, and on the right subject. But it does not answer the question.
A concrete pattern: the user asks 'What is the cheapest flight from JFK to SFO under 6 hours?' The retriever pulls 12 flight listings. The model returns 'There are several flight options between JFK and SFO. Some are operated by JetBlue, others by Delta, and prices range widely depending on the time of booking and class of service. Many flights complete the journey in under 6 hours...', accurate, grounded, completely useless as an answer.
Why it happens mechanistically
The model is autoregressive. Its next-token prediction is most strongly conditioned on the tokens immediately before the prediction point. In the assembled prompt above, the tokens immediately before the answer position are the last few retrieved chunks. The recency-dominant attention is fully captured by retrieval text. The original question lives 5,000+ tokens upstream, in a position where attention has decayed.
The model is not 'ignoring' the question, it processed those tokens during the forward pass. The question's influence on the generation step has decayed because generation is local and recency-weighted.
Why this scales with context length
The longer the retrieval block, the worse the effect. At 1K tokens of retrieval, the question is barely upstream and recency still covers it. At 20K tokens, the question is effectively in the lost-in-the-middle band. The on topic but off question failure rate rises roughly monotonically with retrieval-block length, which is exactly the regime where teams are trying to do better grounding.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's prompt engineering guide for Claude explicitly recommends placing the user query both before and after the document content for long-document QA.
- OpenAI's RAG documentation suggests structuring the prompt as instructions, then context, then question (the question last, by design).
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat happens if you restate the question multiple times throughout the prompt?
Diminishing returns past two placements (top and bottom). A third restate in the middle adds tokens without measurable accuracy gain, because the middle-position content is still in the weak-attention band. The U-shape does not become a W just because you add a middle copy.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Placing the user question only at the top of a long RAG prompt, then watching the model summarize the retrieved chunks instead of answering the actual question.
60 second bullets to scan on the way to the call.
Why placing the question only at the top fails on long contexts
How recency bias makes the bottom of the prompt the strongest-attention slot
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.