Anything the model emits bills as completion_tokens; anything you send (schemas, prior messages, tool results) bills as prompt_tokens on the turn it arrives in the request body.
Picture a restaurant where you pay separately for the menu pages you bring in and for the dishes the chef cooks. Every time you walk back in, the whole stack of menus you carry counts as input, even if some of those pages are notes from the last visit. The chef's plates are output. Now imagine the chef sometimes hands you a sticky note saying 'go grab this ingredient from the pantry' instead of cooking. That sticky note is still something the chef made, so it bills like a dish. When you return with the ingredient, the ingredient and all the prior menus and sticky notes count as input again. Each visit gets its own bill, and the day's total is the sum of all visits, not just the last one.
Detailed answer & concept explanation~7 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
3 min: per-turn input vs output rule + how schemas, tool_calls, and tool results map onto it + cumulative cost across a loop + prompt caching as the natural lever to dampen the quadratic growth.
Real products, models, and research that use this idea.
- An OpenAI Realtime API tool-using assistant in 2026 that exposes 12 functions ships ~6 KB of schemas on every turn, often the largest single contributor to per-call input cost.
- LangChain and the OpenAI Agents SDK both expose per-step token counters precisely because cumulative agent cost is opaque from the final usage block alone.
- Anthropic's Computer Use sessions show the same accounting pattern: screenshot tool results bill as input on the turn they are posted back, often dominating per-step input.
- OpenAI prompt caching (introduced 2024, refined through 2026) targets the schemas + system-prompt prefix specifically because it is the most repeated input chunk in tool-using workloads.
- Production dashboards at Cursor and Cognition's Devin aggregate usage across every step of a coding-agent trajectory rather than reading the last response's usage object.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you reduce the cumulative token cost of a 10-step OpenAI agent loop with a 4 KB tool schema?
QWhy does prompt-token growth across an agent loop look quadratic rather than linear?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Assuming the tool_calls JSON the model emits is free or input-priced, or treating only the final assistant text as the billed output. Both halves of the round-trip bill at the standard input and output rates.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.