A product routes every user query, including trivial factual lookups like "capital of France", through a reasoning model with a 32k thinking token budget and no difficulty based routing. Average chat queries previously took 800 ms on a standard LLM. What happens to p50 latency and cost per query compared to the standard LLM baseline?
Routing easy factual queries through a reasoning model with a large fixed thinking budget raises p50 latency and per-query cost sharply while accuracy barely improves — classic overthinking.
Imagine asking a calculator the capital of France, but it first writes a five-page essay about geography before answering. The answer is still Paris, but you waited much longer and paid for pages you did not need. Reasoning models do something similar when every query gets a huge thinking budget. Hard puzzles benefit from extra steps; simple lookups do not. Without routing easy questions to a fast chat model, you burn time and money for almost no gain.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Reasoning models buy accuracy on hard problems by spending extra tokens at inference. That trade only makes sense when marginal thinking tokens convert into measurable correctness gains. Routing every query — including trivial factual lookups — through a large fixed thinking budget is one of the most common serving mistakes in 2026 reasoning deployments.
This walkthrough explains what happens to p50 latency, per-query cost, and accuracy when easy prompts hit a reasoning model with no difficulty-based routing, and what production teams do instead.
Why reasoning models inflate latency on easy prompts
A standard chat LLM maps input to output in one forward pass per generated token. For "capital of France," the model emits a short answer — often under 20 tokens — and the ~800 ms baseline in the scenario reflects that short decode path plus network and queue time.
A reasoning model adds a thinking phase before the user-visible answer. The policy may allow thousands or tens of thousands of thinking tokens. Each token is generated autoregressively: token N+1 cannot start until token N finishes. Wall-clock latency grows roughly linearly with total generated tokens, modulo batching and speculative decoding.
Even if the model only uses 500 thinking tokens on an easy question — far below a 32k cap — that is still hundreds more decode steps than a chat model needs. p50 latency shifts from sub-second to multi-second. p99 stretches further when some queries trigger longer internal monologues.
Key insight: the budget is an upper bound, not the typical spend — but the serving policy and model habit still push easy queries well past chat latency because nothing tells the model to stop early.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI o-series APIs expose reasoning-effort levels so products cap thinking on simple tasks instead of always using high effort.
- DeepSeek-R1 distilled variants target lower latency for deployment while full R1 retains long-CoT for hard math.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you build a difficulty router without adding noticeable routing latency?
Consider lightweight classifiers on query length, domain keywords, and cheap model confidence scores before invoking the reasoner.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming more thinking tokens always improve every query — easy prompts hit diminishing returns fast and inflate latency without moving accuracy.
60 second bullets to scan on the way to the call.
Define overthinking in one sentence
Explain why p50 latency rises when thinking tokens precede the answer
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.