The right answer is **B**. The 30 percent gap is what the API sees but you do not directly type.
Three invisible contributions account for it.
**1. Chat-template wrappers.** The OpenAI chat completions API wraps each message in role markers. The exact bytes the model sees include something like `<|im_start|>system\n{your system prompt}<|im_end|>\n<|im_start|>user\n{your message}<|im_end|>\n`, plus a BOS at the start. Each wrap adds 4-8 tokens per message. For a 2-message exchange, that is 8-16 extra tokens.
**2. Rendered tool/function schemas.** If your call declares tools (functions, tool_choice), the API renders those schemas as structured text in the prompt. A trivial tool with name and description: ~50 tokens. A non-trivial tool with parameters: 200-500 tokens. A 10-tool agent: 1,000-3,000 tokens easily. These render on every call, not amortized.
**3. Prior conversation turns.** In multi-turn flows, all prior messages get re-sent on every call. By turn 5, input tokens are several times what you typed in the current turn.
**How to measure it correctly.** Either run `tiktoken.get_encoding('o200k_base')` on the exact bytes the API will see (rendered chat template + tool schemas + history), or read `response.usage.prompt_tokens` after the call and reconcile against your estimate.
Options A, C, D are not how billing works. Temperature does not consume tokens. Embeddings are not billed separately on chat completions. OpenAI does not pad token counts.
**B is correct.** The 30 percent gap is the visible versus billed overhead from three sources.
**1. Chat-template wrappers.** The chat completions API renders messages with role markers (`<|im_start|>system`, `<|im_end|>`, equivalents for user and assistant) plus a BOS. Each message wrap is 4-8 tokens. A typical 2-message exchange adds 10-15 tokens of wrapper alone.
**2. Rendered tool/function schemas.** Declared tools serialize into the prompt as JSON-like schema text on every call. A single tool with parameters runs 200-500 tokens; a 10-tool agent can easily add 1,000-3,000 tokens of schema. Tools are not amortized across calls.
**3. Prior conversation turns.** Multi-turn flows re-send all prior messages on every call. Turn N input is the sum of N-1 prior turn pairs plus the current message plus the system plus the tools.
**How to fix the estimation.** Two options. First, run `tiktoken.get_encoding('o200k_base')` on the **exact bytes the API will see**, including the rendered chat template wrappers, the serialized tool schemas, and any prior conversation history. Second, after the call, read `response.usage.prompt_tokens` for the actual billed count and reconcile against your local estimate. Persistent drift indicates a missed factor (often the tool schema overhead).
**Why the wrong options fail.** A (model inflates billing): not how it works; usage matches actual tokenization. C (sampling parameters as control codes): temperature and top_p are server-side parameters, not in the prompt. D (embeddings counted separately): embeddings are a separate API with their own billing; they are not added to chat-completion input token counts.
**Production discipline:** count what the API sees, not what your user types. The 30 percent overhead is normal and predictable once you account for it.
Quick reference
TL;DR
The 30 percent gap is chat-template wrappers (role markers, BOS) plus rendered tool/function schemas plus prior conversation turns. Count the EXACT bytes the API sees, not just the user-visible strings.
Imagine mailing a letter. You weigh the paper inside and expect to pay for that weight. But the post office also weighs the envelope, the address labels, the stamps, and any attached return-receipt cards. Your bill is for the whole package, not just the letter inside. Same with LLM token counts. The user-visible system prompt and user message are the paper. The chat-template wrappers (role markers like 'system:' and 'user:') are the envelope. Declared tools and functions are extra cards inside. Prior conversation turns are previous letters in the same envelope. All of it counts. To predict the bill, you have to count what is actually in the envelope, not just the paper.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
An embeddings-only API call where there is no chat template and no tools. The embeddings API counts only the input text; the chat-template overhead does not apply.
A single-turn no-tools chat completion where the overhead is small and constant. The naive user-visible estimate is close enough to the actual billed count; the full discipline of measuring overhead matters more in tool-heavy or multi-turn settings.
A research script where exact billing accuracy does not matter. Counting just the user-visible content is fine for back of the envelope research math; the discipline applies once the count drives cost decisions.
Quick scan · 2–4 min
Everything important, quickly.
Open with the framing that the user-visible prompt is what you typed; the API sees more. Walk the three invisible contributions: chat-template wrappers (4-8 tokens per message), rendered tool schemas (200-500 per tool, dominates the overhead), conversation history (re-sent every turn). Cover the reconciliation discipline: tiktoken on the exact bytes plus response.usage.prompt_tokens reconciliation. Close on prompt caching as the production lever for static prefixes.
Code example
python
import tiktoken
from openai import OpenAI
client = OpenAI()
enc = tiktoken.get_encoding('o200k_base')
sys = open('system_prompt.md').read()
user_msg = 'What is the refund policy?'
# Naive estimate: just user-visible content
naive_input = len(enc.encode(sys + user_msg))
# Actual API call
resp = client.chat.completions.create(
model='gpt-5.5',
messages=[
{'role': 'system', 'content': sys},
{'role': 'user', 'content': user_msg},
],
tools=my_tool_defs, # this counts too
)
actual_input = resp.usage.prompt_tokens
print(f'naive: {naive_input}, actual: {actual_input}')
# Drift = chat-template wrappers + tool schemas
# If you forget the tools in your estimate, drift is much larger
Real-world examples
Real products, models, and research that use this idea.
1OpenAI's chat completions response.usage object returns prompt_tokens, completion_tokens, and cached_tokens; these are the billed counts and the only authoritative numbers for cost reconciliation.
2Anthropic's messages.count_tokens API accepts tools as a parameter and returns input_tokens that includes the rendered tool schema overhead.
3Production agent frameworks (LangGraph, LlamaIndex) expose per-call token tracking that decomposes input into system, user, history, and tool components for debugging.
4Prompt-caching pricing pages from OpenAI explicitly document that long static system prompts hit the cache at half-rate, making cache-hit ratio one of the main cost-control levers for production deployments.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf you reduce the system prompt by 200 tokens, how much do you actually save per call?
Sign in or go Pro to see the approach for this follow-up.
QYour tool count grew from 3 to 12 over a release. The bill rose 40 percent and you have no other changes. Diagnose.
Sign in or go Pro to see the approach for this follow-up.
QHow would you architect a high-QPS service to minimize per-call token overhead?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Estimating token count from the user-visible system prompt plus user message alone, missing the chat-template wrappers and rendered tool schemas that the API charges for.