Why can a 50 token reply from a reasoning model still bill you for thousands of tokens?

Short answer·Medium·4.0 · 0·~3 min·Asked atModal Labs Netflix OpenAI

Attempt it

Explain how reasoning models can charge for tokens the caller never sees in the response. Cover what those hidden tokens are, why they cost real money, and how providers typically surface them in usage accounting.

Free · 2 AI evals / day

**Reasoning models emit a hidden chain-of-thought.** Systems like the OpenAI o-series, Anthropic Claude Opus 4.7 with extended thinking, and DeepSeek V4 are trained to generate internal reasoning tokens before they produce the visible answer. Those reasoning tokens are real generated tokens, sampled by the same decode loop as the final reply. Each reasoning token costs the same GPU time as a visible token. The forward pass, the KV-cache read from HBM, the matmul over the parameters: all of it runs identically. The model just chooses (because of how it was trained) to mark some output as reasoning and some as the final answer, and the serving runtime hides the reasoning from the response payload by default. Providers bill for the GPU work, so reasoning tokens bill as output (completion) tokens, at the higher of the two pricing tiers. The standard pattern is to surface them in the usage object: OpenAI uses `completion_tokens_details.reasoning_tokens`, DeepSeek uses `thinking_tokens`, Anthropic uses an extended-thinking counter. Total billed output equals visible completion tokens plus reasoning tokens. Two practical implications. First, set output budget caps (`max_completion_tokens` or equivalent). Without one, a single request can run for tens of thousands of reasoning tokens and cost orders of magnitude more than you expected. Second, when comparing reasoning models to non-reasoning ones on cost, always compare total billed completion tokens, not the visible answer length, or you will reach the wrong conclusion.

**The hidden tokens are a chain-of-thought, and they bill as output.** Reasoning-tuned models (OpenAI o-series, Claude with extended thinking, DeepSeek V4) generate a long internal reasoning trace before producing the visible answer. Each reasoning token is a normal decode step: full forward pass, full KV-cache read, full HBM bandwidth load. The serving runtime hides the trace from the response payload, but the GPU did the work. Providers bill for the GPU work. Reasoning tokens count as completion (output) tokens, at the same price as visible output. Total billed output equals visible_tokens plus reasoning_tokens, and the reasoning side typically dominates: a 50-token visible reply can hide 5,000 reasoning tokens behind it, so the bill is 100x what the visible length would suggest. Usage objects surface this. OpenAI exposes `completion_tokens_details.reasoning_tokens`. DeepSeek exposes a `thinking_tokens` counter. Anthropic exposes the same idea under extended-thinking accounting. Production dashboards that only watch `completion_tokens` undercount badly once a workload moves to a reasoning tier. Two practical defenses. **Set output-side budget caps** like `max_completion_tokens` or an explicit reasoning-token cap; without one, a single request has no upper bound on cost. **Normalize cost comparisons on total billed completion tokens**, not visible answer length, when picking between a reasoning model and a non-reasoning one. The reasoning model often loses on raw cost-per-task once the hidden tokens are counted honestly. Whether the higher cost is worth it is a separate question, but it should be answered with the right numbers.

TL;DR

Reasoning models generate a hidden chain-of-thought before the visible answer. Those tokens cost real GPU decode time and bill as output tokens, often dwarfing the visible reply by 10-100x.

Memory aid

Easy to grasp

Picture a student who scribbles three pages of working notes to figure out a math problem and then writes a one-line answer on the test sheet. The teacher only sees the one line, but if they were paying the student per page of work produced, they would owe for the notes too. A reasoning model is the same. Before it writes the final answer you see, it generates a long internal scratchpad of partial reasoning, hypothesis checks, and self-corrections. The provider's GPU did real work for every one of those scratchpad tokens, so the bill includes them. The visible reply might be 50 tokens, but the line item on your invoice can show 5,000 output tokens because the model wrote 4,950 tokens worth of hidden thinking to get there.

Key concepts

Chain of thought Reasoning models Token pricing

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Reasoning models broke the implicit contract between visible output length and bill size. For a decade of API serving, completion_tokens was a reasonable proxy for cost. With the rise of reasoning-tuned models in 2024 and 2025, that proxy collapsed; the visible answer became almost meaningless as a cost predictor because the bulk of generated tokens now live in a hidden chain-of-thought pass.

This question tests whether you understand the mechanism (real decode tokens, real GPU work, billed as output), the observability surface (usage-object fields that expose the counter), and the operational defenses (budget caps, difficulty routing, comparison hygiene). A strong answer treats reasoning tokens as a budget line item that needs its own dashboard and its own cap, not as model behavior to be ignored.

The deep dive walks the mechanism end to end, explains how the bill surfaces the cost, lays out the observability stack, and gives the mitigation playbook in the order it typically pays off in production.

What reasoning tokens actually are

A reasoning-tuned model is trained, usually with reinforcement learning on a reward that scores final-answer correctness, to emit an internal chain-of-thought before the user-visible reply. The internal thoughts are real generated tokens drawn from the same vocabulary, sampled by the same decode loop, and streamed through HBM at the same per-step cost as any other generation.

The model itself does not distinguish thinking tokens from answer tokens at the architecture level; the boundary is learned. DeepSeek V4 uses <think> and </think> tags. OpenAI's o-series wraps reasoning in opaque segments that the API summarizes. Anthropic's extended thinking uses a structured thinking block returned in the API response (or hidden by default).

The serving runtime usually strips or hides the thinking region from the visible UI by default. But it can and does include them in the API response when the caller asks, and it always includes them in the usage accounting.

Thinking tokens are output tokens by every metric that matters operationally. They consume KV cache, they pay decode-step latency, they show up on the bill. The UI's choice to hide them is presentation, not accounting.

How the bill surfaces them

Why amplification is heavy-tailed

The defense playbook

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

OpenAI o-series exposes reasoning tokens in completion_tokens_details.reasoning_tokens and bills them at the standard output rate.
Anthropic Claude Opus 4.7 with extended thinking exposes a budget_tokens parameter and surfaces reasoning tokens in the usage object alongside output_tokens.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you instrument cost monitoring to catch reasoning-token regressions early?

Track reasoning_tokens per route, per intent class, and per customer. Alert on the reasoning-to-completion ratio crossing a threshold week over week. Sample the actual hidden traces for the worst-amplification routes to see what the model is over-thinking.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Comparing reasoning models to non-reasoning models on the length of the visible answer. The reasoning model looks cheap by that metric and ruinously expensive on the actual bill, because thinking tokens dominate output count.

60 second bullets to scan on the way to the call.

What reasoning tokens are and how they enter the decode loop
Why they bill as output (completion) tokens

Primary sources. Browse if you want the original framing.