Tool output belongs immediately before the model's next reasoning step, at the recency end of the context. The standard tool-use formats from Anthropic, OpenAI, and Google all encode this naturally: the tool_use message is followed by a tool_result message, and the assistant's next response sits right after. Placing tool results at the top of the conversation or in a 'pinned' system block puts them in the middle of a long context where the lost-in-the-middle effect kicks in. The native message ordering exists precisely because recency placement is what works.
When a tool call finishes, its output needs to land somewhere in the context before the next model turn. The right place is **immediately before the next reasoning step**, at the recency end of the conversation.
The reason is how transformer attention behaves over long contexts. The **lost-in-the-middle** effect, documented across all modern LLMs, says attention is strongest at the very start of the context and at the recency end, and weakest in the middle. The model attends to what is in front of it; a tool result placed at the top of a long conversation gets pushed into the middle trough as more turns accumulate, and effectively vanishes.
Placing the tool output just before the next assistant turn solves this naturally. It sits in the strong-attention recency region, where the model is about to reason. The native message formats from Anthropic, OpenAI, and Google all encode this: a tool_use message from the assistant is followed by a tool_result message carrying the output, and then the assistant's next response. The ordering is not arbitrary; it is the encoding that the providers designed around the empirical observation that this is what reliably works.
The failure mode is to treat tool outputs as global pinned state, sticking them at the top, dumping them in the system prompt, or keeping a running results dictionary at the start of context. All of these put tool outputs into a position where attention is weakest by the time the model needs them.
Tool outputs belong at the **recency end of context**, immediately before the assistant's next turn. This is the position the native tool-use formats from Anthropic, OpenAI, and Google all use, and the reason is the lost-in-the-middle attention curve.
The attention pattern across long contexts is U-shaped. Tokens near the absolute start of the context get strong attention (the primacy slot), tokens near the recency end get strong attention (the recency slot), and tokens in the middle get attention-starved. This shape is consistent across every modern frontier LLM and is the load-bearing finding behind a lot of context-engineering practice.
The natural place for a tool result is the recency slot because that is where the model is about to reason next. Placing the result anywhere else creates a distance between the observation and the reasoning that consumes it, and as more turns accumulate, that distance grows. A tool output at the top of a 30-turn conversation is no longer at the top of a small context, it is buried in the middle of a large one, in the deep trough of the attention curve.
The native tool-use encoding from each major provider is the same pattern. The assistant produces a tool_use message. The runtime appends a tool_result message carrying the output. The assistant's next message reads the tool_result and produces a response. This ordering keeps the result immediately before its consumer. Any custom design that reorders this, pinning tool outputs at the top, collecting them in a sidebar, sticking them in the system prompt, fights the format that was built around the empirical finding.
The practical implication for code that builds prompts manually (not all systems use the native message API): the conversation history list should keep tool results inline with the turn that produced them, never hoist them out.
Quick reference
TL;DR
Tool results go right before the assistant's next reasoning turn, at the recency end of context, which is exactly where the native tool-use message formats from Anthropic, OpenAI, and Google place them.
Think about a relay race where one runner hands a baton to the next. If the first runner drops the baton at the starting line and the second runner is at the finish, the handoff fails, the baton is too far away to be useful. A tool result is the baton, and the next model turn is the runner waiting for it. The handoff has to happen at the exact spot where the next runner is standing. Placing tool output at the top of a long conversation is like leaving the baton back at the starting line; the next runner cannot see it through everything that has happened in between. The native chat format for tool use is just a relay rule: hand the baton off right at the spot where the next leg of the race begins.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Sessions short enough that the entire trajectory fits in the primacy region. If the whole conversation is under a few thousand tokens, placement matters much less. The lost-in-the-middle effect only becomes load-bearing once the trajectory grows.
Single-shot non-agent flows where there is one tool call and one response with nothing in between. With no trajectory, there is no middle to be lost in. Placement is degenerate when only one turn exists.
Forensic replay or evaluation harnesses that need to test how the model handles specific prompt layouts. When the goal is to study a specific (possibly suboptimal) layout, you intentionally deviate from the production placement rule to measure what breaks.
Quick scan · 2–4 min
Everything important, quickly.
State the answer: tool result goes at the recency end, immediately before the next assistant turn. Walk through the lost-in-the-middle attention curve and explain why recency is the durably strong zone. Tie this to the native tool-use message formats of Anthropic, OpenAI, and Google, which encode the placement by construction. Cover the failure modes: pinning at the top, stashing in the system prompt, building a custom tool_results dictionary. Close with the cross-turn recall pattern: durable observations live in agent state with a recall tool, not in pinned prompt regions.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's Claude API uses a tool_use block from the assistant followed by a tool_result block carrying the output, immediately before the next assistant response.
2OpenAI's Responses API places tool_calls as assistant message parts with paired tool messages right after, threading them inline with the conversation.
3Google's Gemini function-calling format pairs function_call and function_response in adjacent turns, never as global pinned state.
4LangGraph stores tool observations in typed state but renders them into the immediate next prompt turn rather than pinning them at the top of context.
5Cursor and Claude Code both thread file-read and shell-command results inline with the reasoning step that uses them, never as a pinned sidebar.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat is the right way to give the model access to old tool results from many turns ago?
Sign in or go Pro to see the approach for this follow-up.
QHow does this principle interact with prompt caching?
Sign in or go Pro to see the approach for this follow-up.
QWhat goes wrong if you place tool outputs *after* the next assistant turn instead of before?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Pinning tool outputs at the top of the conversation in the belief that primacy gives them priority, in a long trajectory that just buries them in the middle.