The 30 percent gap is chat-template wrappers (role markers, BOS) plus rendered tool/function schemas plus prior conversation turns. Count the EXACT bytes the API sees, not just the user-visible strings.
Imagine mailing a letter. You weigh the paper inside and expect to pay for that weight. But the post office also weighs the envelope, the address labels, the stamps, and any attached return-receipt cards. Your bill is for the whole package, not just the letter inside. Same with LLM token counts. The user-visible system prompt and user message are the paper. The chat-template wrappers (role markers like 'system:' and 'user:') are the envelope. Declared tools and functions are extra cards inside. Prior conversation turns are previous letters in the same envelope. All of it counts. To predict the bill, you have to count what is actually in the envelope, not just the paper.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Every production team using OpenAI or Anthropic eventually notices that response.usage.prompt_tokens is meaningfully higher than the local count of the user-visible system prompt plus user message. The gap is reliably 20-40 percent for typical agent calls and can hit 100 percent or more for tool-heavy multi-turn workloads. Understanding where the gap comes from is the difference between a credible cost model and a bill that surprises finance.
The right answer is B: the gap is chat-template wrappers plus rendered tool schemas plus conversation history. None of these are visible in the developer-visible prompt content, but all three are part of the bytes the API sees and bills for. The other three options are common confusions: A (model inflates billing) is conspiracy-thinking; C (sampling parameters consume tokens) confuses request parameters with prompt content; D (embeddings counted separately) confuses two different APIs.
The rest of this explanation walks each of the three real contributors in detail, names the production tactics for measuring and minimizing the overhead, and explains why each of the wrong options misdiagnoses the cost structure.
Contributor 1: chat-template wrappers and BOS
The OpenAI chat completions API does not send your messages as plain text. It renders them into a specific format the model was trained on, with role markers and special tokens that mark the boundaries of each message. The exact format depends on the model family but typically looks like:
<|im_start|>system\n{your system prompt}<|im_end|>\n
<|im_start|>user\n{your user message}<|im_end|>\n
<|im_start|>assistant\n
The role markers (<|im_start|>, <|im_end|>) plus the BOS plus the newlines add 4-8 tokens per message. A 2-message exchange (system + user) adds roughly 10-15 tokens of wrapper. A 5-turn multi-turn flow (system + 5 user/assistant pairs) adds 50-80 tokens of wrappers alone, all billed as input.
The wrappers exist because the model was instruction-tuned to recognize them as turn boundaries. The model uses these tokens to parse the conversation structure. They are not metadata the API could omit; they are content the model has to see.
A local estimator that counts only the user-visible content misses these tokens entirely. The fix is to render the chat template yourself (most models expose apply_chat_template) before counting, or to add a fixed per-message overhead estimate (4-8 tokens per message) on top of the user-visible count.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import tiktoken
from openai import OpenAI
client = OpenAI()
enc = tiktoken.get_encoding('o200k_base')
sys = open('system_prompt.md').read()
user_msg = 'What is the refund policy?'
# Naive estimate: just user-visible content
naive_input = len(enc.encode(sys + user_msg))
# Actual API call
resp = client.chat.completions.create(
model='gpt-5.5',
messages=[
{'role': 'system', 'content': sys},
{'role': 'user', 'content': user_msg},
],
tools=my_tool_defs, # this counts too
)
actual_input = resp.usage.prompt_tokens
print(f'naive: {naive_input}, actual: {actual_input}')
# Drift = chat-template wrappers + tool schemas
# If you forget the tools in your estimate, drift is much largerReal products, models, and research that use this idea.
- OpenAI's chat completions response.usage object returns prompt_tokens, completion_tokens, and cached_tokens; these are the billed counts and the only authoritative numbers for cost reconciliation.
- Anthropic's messages.count_tokens API accepts tools as a parameter and returns input_tokens that includes the rendered tool schema overhead.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf you reduce the system prompt by 200 tokens, how much do you actually save per call?
If the system prompt is cached (long, static, hits OpenAI's prompt cache), you save the discounted cached rate, which is roughly half input price. If the system prompt is dynamic (changes per request) and cache-misses, you save the full input rate. Confirm via response.usage.prompt_tokens_details.cached_tokens to see how much of the prompt was served from cache before optimizing.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Estimating token count from the user-visible system prompt plus user message alone, missing the chat-template wrappers and rendered tool schemas that the API charges for.
60 second bullets to scan on the way to the call.
Three sources of invisible but billed tokens: chat template, tools, history.
Approximate token count per chat-template wrap (4-8).
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.