Estimate the token tax a single tool round trip adds compared with an inline answer

Predict output·Medium·4.0 · 0·~2 min·Asked atIntel OpenAI Samsung·Relevant atAnthropic

Attempt it

A chat app exposes one tool (`get_weather`) with a 180-token JSON schema. A user asks 'what is the weather in Tokyo?' (8 input tokens, system prompt 50 tokens). Two paths:

Path A, inline answer (no tools): model replies directly with a 25-token guess.

Path B, function calling round trip:
  Turn 1: prompt includes system + tool schemas + user message; model emits a `tool_call` message (40 output tokens: function name + JSON args).
  Tool executes externally (no model cost).
  Turn 2: prompt now includes everything from Turn 1 plus the tool_call assistant message plus a 60-token tool_result message; model emits the final 30-token natural language answer.

Using OpenAI style accounting where the full conversation is re-sent on every turn, compute:
  1. Total billed input tokens across Path B (sum across both turns)
  2. Total billed output tokens across Path B
  3. The Path B / Path A token cost ratio (assuming equal input and output unit price)

Report all three numbers.

Function calling makes the API conversational. The model asks for a tool by emitting a structured tool_call message, your code runs the tool externally, and then you call the model again with the tool's result so it can write the final answer. That second call has to include everything the first call had plus the tool exchange, because the model is stateless between API calls and needs the full conversation to ground its answer. Let us count tokens for the given example. Path A is the inline answer: 50 tokens of system prompt, 8 tokens of user question, 25 tokens of model output. Total 83 tokens. Path B starts the same way. Turn 1 input is 50 (system) + 180 (the tool schema describing get_weather) + 8 (user question) = 238 tokens. Output is the tool_call: 40 tokens of structured JSON (function name plus arguments). The tool runs externally and returns 60 tokens of result; this does not cost anything because the model did not run. Now Turn 2. The API call has to include the entire conversation again. That is the same 50 system + 180 schema + 8 user, plus the 40-token assistant tool_call message, plus the 60-token tool_result message. Input is 338 tokens. The model writes the final answer: 30 output tokens. Add up Path B. Input across both turns: 238 + 338 = 576. Output across both turns: 40 + 30 = 70. Total 646 tokens. Compare to Path A's 83 tokens. The ratio is 646 / 83, about 7.8 times more expensive. And this is the simple case with one small tool. The same arithmetic with five tools, or with chained tool calls, blows up much further. The standard mitigation is prompt caching on the schema prefix. Both Anthropic and OpenAI let you mark the system prompt and schemas as cacheable; the second call pays a small discounted rate for those tokens instead of the full input price. That alone cuts most of the tool-use tax for steady traffic.

Tool-calling overhead is real and almost entirely on the input side. The token-accounting model that every major provider uses (OpenAI, Anthropic, Google) treats the API as stateless across calls: every turn re-sends the full conversation including system prompt, tool schemas, user messages, prior assistant messages, and any tool_result messages. The model is recomputed against the entire sequence on each call. Walking the example: Path A is one call, 50 + 8 = 58 input tokens, 25 output. Total 83 tokens. Path B requires two model calls plus the tool execution between them. Turn 1 input is 50 (system) + 180 (schema) + 8 (user) = 238 input tokens. Output is the structured tool_call: 40 tokens. Tool runs externally, returns 60 tokens of result, which incurs no model cost. Turn 2 is where the cost compounds. The conversation now includes the original system, schema, and user message PLUS the assistant's prior tool_call message PLUS the tool_result message. All of that re-enters the input window: 50 + 180 + 8 + 40 + 60 = 338 input tokens. Final output is 30 tokens. Totals: Path B input = 238 + 338 = 576. Path B output = 40 + 30 = 70. Total Path B = 646 tokens. Ratio against Path A: 646 / 83 ≈ 7.8x. Notice what is dominating the bill. Input tokens are 576 out of 646, almost 90 percent of the spend. The user's natural-language answer (30 tokens) is less than 5 percent. Most of the cost is paying twice for the schema and once each for the tool_call and tool_result messages. This scales badly. Add a second tool: the schema cost doubles in both turns. Add a multi-step agent that takes 5 tool calls: now you have 5 turns, each re-sending the schema and the full prior tool history, which goes from N^2 cost in the worst case. The schema is paid on every turn even though it does not change between turns. Production mitigations layer in this order. First, prompt caching.

TL;DR

A two-turn tool round-trip re-sends the schema and the prior assistant + tool messages on Turn 2, so a one-tool workflow with a 180-token schema costs about 7.8x what an inline answer would, dominated by input

Memory aid

Easy to grasp

Imagine asking a librarian a question. Inline answer is just talking: you ask, they answer. Tool-calling is more like writing letters. First letter: you send the question plus the rulebook of tools the librarian can use, and the librarian writes back saying 'I need to use the weather tool, give me Tokyo.' Then someone fetches the answer. Second letter: you send the original question AND the rulebook AGAIN AND the librarian's request AND the tool's reply, so the librarian has the full story to write the final answer. You paid postage for the rulebook twice and for the conversation history once. That is why a simple question with one tool can cost eight times what a direct answer would.

Key concepts

Function calling Tool use Tool schema Prompt caching Input and output pricing

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Function calling looks free in the SDK but is paid for in tokens, every turn. The API contract is stateless: each call re-bills the entire conversation as input, which means the schema and the prior tool exchange are paid on every subsequent turn. A simple one-tool round-trip already costs roughly 7-10x what the inline answer would, and the cost compounds badly in multi-step agents.

This deep dive walks through the token accounting on the example, generalizes to the agent case, and connects each cost component to its production mitigation.

Token accounting on Path A versus Path B

Path A is the simplest possible flow: one API call, model answers directly. Input = 50 (system) + 8 (user) = 58 tokens. Output = 25 tokens. Total = 83 tokens.

Path B requires two API calls plus an external tool execution.

Turn 1 input: the system prompt (50), the tool schema (180), and the user message (8). Total input = 238 tokens. The model emits a structured tool_call message: 40 output tokens (function name plus JSON args). The API responds, the calling code parses the tool_call and runs the tool externally. The tool returns 60 tokens worth of content (e.g., 'Tokyo, 14C, light rain, humidity 78%'). The tool execution itself is not billed to the model.

Turn 2 input: the same system, schema, and user message PLUS the prior assistant tool_call (40 tokens) PLUS the tool_result (60 tokens). Total = 50 + 180 + 8 + 40 + 60 = 338 tokens. The model finally writes the natural-language answer: 30 output tokens.

Add it up. Path B input across both turns: 238 + 338 = 576. Path B output: 40 + 30 = 70. Total: 646 tokens. Ratio against Path A's 83: 7.78x.

Look at where the cost actually lives. Of those 646 tokens, only 30 are the final user-facing answer. 70 (less than 11 percent) are model output total. 576 (almost 90 percent) are input. The single largest line item is the schema, paid twice for a total of 360 tokens, which is more than the actual answer the user gets back.

Generalizing to N-step agents

Prompt caching is the most important single mitigation

Schema minimization and tool granularity

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Token component	Path A (inline)	Path B (tool)	Notes
System prompt	50 in	50 + 50 = 100 in	Paid every turn
Tool schema	0	180 + 180 = 360 in	Paid every turn (cacheable)
User message	8 in	8 + 8 = 16 in	Paid every turn
Tool_call message	0	40 out + 40 in (Turn 2)	Output Turn 1, input Turn 2
Tool_result message	0	60 in	Input on Turn 2
Final answer	25 out	30 out	Tokens the user actually sees
Totals	58 in + 25 out = 83	576 in + 70 out = 646	7.8x ratio

Real products, models, and research that use this idea.

Anthropic Claude tool use with explicit cache_control on the tools array routinely cuts tool-using-workflow costs by 4-8x for steady traffic, by reducing the schema re-input to a small fraction of the full input rate.
OpenAI's function calling auto-caches the system prompt and tools above 1024 tokens, applied automatically to recent prefix repeats; this is why high-volume tool users see lower realized prices than the published rate card.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the ratio change as the schema grows to 1000 tokens or shrinks to 50?

The schema is paid on every turn. Doubling its size doubles its contribution to the bill. The ratio versus Path A scales linearly with schema size at fixed turn count. Trimming schema verbosity is one of the highest-leverage optimizations for tool-using products.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Counting only the final answer tokens. The big cost is input: the schema is in the prompt twice (once per turn), and the tool_call and tool_result are re-fed as input on Turn 2. Plan capacity in tokens, not requests.

60 second bullets to scan on the way to the call.

Explain what each API call re-sends on every turn.
Describe where tool schemas appear in the input.

Primary sources. Browse if you want the original framing.