Your GPT-5.5 call bills 30 percent more tokens than the user prompt plus system instruction. Where are the invisible tokens?

MCQ·Medium·4.0 · 0·~1 min·Relevant atOpenAI

Attempt it

The right answer is **B**. The 30 percent gap is what the API sees but you do not directly type. Three invisible contributions account for it. **1. Chat-template wrappers.** The OpenAI chat completions API wraps each message in role markers. The exact bytes the model sees include something like `<|im_start|>system\n{your system prompt}<|im_end|>\n<|im_start|>user\n{your message}<|im_end|>\n`, plus a BOS at the start. Each wrap adds 4-8 tokens per message. For a 2-message exchange, that is 8-16 extra tokens. **2. Rendered tool/function schemas.** If your call declares tools (functions, tool_choice), the API renders those schemas as structured text in the prompt. A trivial tool with name and description: ~50 tokens. A non-trivial tool with parameters: 200-500 tokens. A 10-tool agent: 1,000-3,000 tokens easily. These render on every call, not amortized. **3. Prior conversation turns.** In multi-turn flows, all prior messages get re-sent on every call. By turn 5, input tokens are several times what you typed in the current turn. **How to measure it correctly.** Either run `tiktoken.get_encoding('o200k_base')` on the exact bytes the API will see (rendered chat template + tool schemas + history), or read `response.usage.prompt_tokens` after the call and reconcile against your estimate. Options A, C, D are not how billing works. Temperature does not consume tokens. Embeddings are not billed separately on chat completions. OpenAI does not pad token counts.

**B is correct.** The 30 percent gap is the visible versus billed overhead from three sources. **1. Chat-template wrappers.** The chat completions API renders messages with role markers (`<|im_start|>system`, `<|im_end|>`, equivalents for user and assistant) plus a BOS. Each message wrap is 4-8 tokens. A typical 2-message exchange adds 10-15 tokens of wrapper alone. **2. Rendered tool/function schemas.** Declared tools serialize into the prompt as JSON-like schema text on every call. A single tool with parameters runs 200-500 tokens; a 10-tool agent can easily add 1,000-3,000 tokens of schema. Tools are not amortized across calls. **3. Prior conversation turns.** Multi-turn flows re-send all prior messages on every call. Turn N input is the sum of N-1 prior turn pairs plus the current message plus the system plus the tools. **How to fix the estimation.** Two options. First, run `tiktoken.get_encoding('o200k_base')` on the **exact bytes the API will see**, including the rendered chat template wrappers, the serialized tool schemas, and any prior conversation history. Second, after the call, read `response.usage.prompt_tokens` for the actual billed count and reconcile against your local estimate. Persistent drift indicates a missed factor (often the tool schema overhead). **Why the wrong options fail.** A (model inflates billing): not how it works; usage matches actual tokenization. C (sampling parameters as control codes): temperature and top_p are server-side parameters, not in the prompt. D (embeddings counted separately): embeddings are a separate API with their own billing; they are not added to chat-completion input token counts. **Production discipline:** count what the API sees, not what your user types. The 30 percent overhead is normal and predictable once you account for it.

Quick reference

TL;DR

The 30 percent gap is chat-template wrappers (role markers, BOS) plus rendered tool/function schemas plus prior conversation turns. Count the EXACT bytes the API sees, not just the user-visible strings.

Memory aid

Easy to grasp

Imagine mailing a letter. You weigh the paper inside and expect to pay for that weight. But the post office also weighs the envelope, the address labels, the stamps, and any attached return-receipt cards. Your bill is for the whole package, not just the letter inside. Same with LLM token counts. The user-visible system prompt and user message are the paper. The chat-template wrappers (role markers like 'system:' and 'user:') are the envelope. Declared tools and functions are extra cards inside. Prior conversation turns are previous letters in the same envelope. All of it counts. To predict the bill, you have to count what is actually in the envelope, not just the paper.

Key concepts

Chat template Tool schema System prompt

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Open with the framing that the user-visible prompt is what you typed; the API sees more. Walk the three invisible contributions: chat-template wrappers (4-8 tokens per message), rendered tool schemas (200-500 per tool, dominates the overhead), conversation history (re-sent every turn). Cover the reconciliation discipline: tiktoken on the exact bytes plus response.usage.prompt_tokens reconciliation. Close on prompt caching as the production lever for static prefixes.

Code example

python

import tiktoken
from openai import OpenAI

client = OpenAI()
enc = tiktoken.get_encoding('o200k_base')

sys = open('system_prompt.md').read()
user_msg = 'What is the refund policy?'

# Naive estimate: just user-visible content
naive_input = len(enc.encode(sys + user_msg))

# Actual API call
resp = client.chat.completions.create(
    model='gpt-5.5',
    messages=[
        {'role': 'system', 'content': sys},
        {'role': 'user', 'content': user_msg},
    ],
    tools=my_tool_defs,  # this counts too
)
actual_input = resp.usage.prompt_tokens

print(f'naive: {naive_input}, actual: {actual_input}')
# Drift = chat-template wrappers + tool schemas
# If you forget the tools in your estimate, drift is much larger

Real-world examples

Real products, models, and research that use this idea.

OpenAI's chat completions response.usage object returns prompt_tokens, completion_tokens, and cached_tokens; these are the billed counts and the only authoritative numbers for cost reconciliation.
Anthropic's messages.count_tokens API accepts tools as a parameter and returns input_tokens that includes the rendered tool schema overhead.
Production agent frameworks (LangGraph, LlamaIndex) expose per-call token tracking that decomposes input into system, user, history, and tool components for debugging.
Prompt-caching pricing pages from OpenAI explicitly document that long static system prompts hit the cache at half-rate, making cache-hit ratio one of the main cost-control levers for production deployments.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf you reduce the system prompt by 200 tokens, how much do you actually save per call?

QYour tool count grew from 3 to 12 over a release. The bill rose 40 percent and you have no other changes. Diagnose.

QHow would you architect a high-QPS service to minimize per-call token overhead?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Estimating token count from the user-visible system prompt plus user message alone, missing the chat-template wrappers and rendered tool schemas that the API charges for.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.