Drag each answer to line up with its matching prompt
Tool / function schemas sent in the request
completion_tokens on the second turn
Assistant message containing tool_calls with JSON arguments
Sum of prompt + completion across every turn, not the last one
Tool result message you POST back on the next turn
prompt_tokens on the turn they are sent
Final assistant text after the tool result
prompt_tokens on the next turn
Cumulative agent loop cost (multi-tool sessions)
completion_tokens on the turn that emitted them
Anything the model emits bills as completion_tokens; anything you send (schemas, prior messages, tool results) bills as prompt_tokens on the turn it arrives in the request body.
Picture a restaurant where you pay separately for the menu pages you bring in and for the dishes the chef cooks. Every time you walk back in, the whole stack of menus you carry counts as input, even if some of those pages are notes from the last visit. The chef's plates are output. Now imagine the chef sometimes hands you a sticky note saying 'go grab this ingredient from the pantry' instead of cooking. That sticky note is still something the chef made, so it bills like a dish. When you return with the ingredient, the ingredient and all the prior menus and sticky notes count as input again. Each visit gets its own bill, and the day's total is the sum of all visits, not just the last one.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Token accounting in a tool-using OpenAI session trips up almost every team the first time they build an agent. The naive read of the usage object in the final response says the session cost 2,000 input and 400 output tokens, when the real session, with seven intermediate tool hops, actually burned closer to 18,000 input tokens. The dashboard understates the bill by an order of magnitude, the finance team is surprised on the next invoice, and the postmortem ends with someone writing total_tokens += response.usage.total_tokens inside the loop.
The whole confusion comes from treating tool calling as something fundamentally new. It is not. The API still bills the same two buckets, prompt tokens for everything in the request body, completion tokens for everything the model generated, applied turn by turn. Tool calling just multiplies the number of turns and rearranges what lives in each one. Get the per-turn rule right and the full session accounting falls out automatically.
This deep dive walks the boundary piece by piece: tool schemas (caller-supplied, repeat on every turn), the assistant's tool_calls JSON (model-generated, billed as output on the turn it appears), the tool-result message (caller-supplied even though it represents the tool's reply, billed as input), and the final assistant text (model-generated, output). It then shows the cumulative shape of an N-step loop and the levers, schema trimming and prompt caching chief among them, that production agents use to keep the bill survivable.
The per-turn billing rule, applied without exceptions
Every OpenAI chat completion call returns a usage object with three numbers: prompt_tokens, completion_tokens, total_tokens. The rule that generates those numbers is mechanically simple: tokenize everything in the request body and call that prompt_tokens; tokenize everything the model writes in the response and call that completion_tokens. There are no other categories.
Tool calling adds more kinds of content to each side of the boundary, not new sides. The tools parameter, function names, descriptions, JSON-schema parameters, is part of the request body, so it bills as input. The messages array, including every prior role: "user", role: "assistant", role: "system", and role: "tool" entry, is also in the request body, so it bills as input too. The response contains an assistant message whose content may be text and may include a tool_calls array; both halves are output.
Applied turn by turn, this rule never breaks. Confusion usually comes from inheriting a model where 'input' means 'user-typed' and 'output' means 'model-generated to the user'. The API does not work that way. The API sees raw request bytes versus raw response bytes.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- An OpenAI Realtime API tool-using assistant in 2026 that exposes 12 functions ships ~6 KB of schemas on every turn, often the largest single contributor to per-call input cost.
- LangChain and the OpenAI Agents SDK both expose per-step token counters precisely because cumulative agent cost is opaque from the final usage block alone.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you reduce the cumulative token cost of a 10-step OpenAI agent loop with a 4 KB tool schema?
Cache the stable prefix (system + schemas) via OpenAI prompt caching to halve its re-input cost. Trim schemas (drop verbose descriptions, narrow parameter enums). Compact or summarize older turns once their full content is no longer needed. Avoid resending tool results once the model has consumed them.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming the tool_calls JSON the model emits is free or input-priced, or treating only the final assistant text as the billed output. Both halves of the round-trip bill at the standard input and output rates.
60 second bullets to scan on the way to the call.
How tool schemas bill on every turn they are sent
Why assistant tool_calls JSON counts as completion_tokens
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.