At inference, reasoning models spend extra decode steps on long internal traces before the answer — from RL-shaped post-training plus a larger thinking budget, not just slower chat sampling.
A chat model is like someone who answers as soon as they open their mouth. A reasoning model is like someone who is allowed to scribble on scratch paper first — sometimes pages of it — then gives the final answer. The scratch paper habit was practiced in special training, and the app gives extra time and space for it. It is not the same person talking more slowly.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The flashcard front asks a deceptively simple question: what changes at inference when you switch from chat to a reasoning model? Candidates who only say "it thinks more" miss the trained policy and serving economics. Interviewers want the full three-part answer.
This walkthrough builds the inference-time picture from generation structure through billing and routing.
Generation structure: thinking before answering
A chat completion is largely answer-shaped: system prompt, user message, assistant tokens. Reasoning completions add a thinking phase — sometimes thousands of tokens — before the outward response. Providers may label these thinking tokens separately in API responses.
The model may backtrack, enumerate cases, or self-critique inside that phase. Users might see it (expanded panel), see a summary, or see nothing — but the GPU still generated those tokens.
Wall-clock latency rises because autoregressive decode is sequential. p50 shifts from sub-second chat to multi-second reasoning on many workloads. This is expected behavior, not a performance bug.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI o-series APIs bill thinking tokens separately from completion tokens with reasoning-effort controls.
- DeepSeek-R1 exposes long CoT before final answers with distinct latency profiles vs DeepSeek-V3 chat.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does KV cache growth differ for 8k thinking tokens vs 200-token chat?
Linear in sequence length per layer — memory and bandwidth dominate reasoning batch planning.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Equating reasoning models with slower chat — the difference is trained long-CoT policy plus thinking-token budget, not sampling speed alone.
60 second bullets to scan on the way to the call.
Name extra decode / thinking phase before final answer
Contrast trained policy vs prompt-only behavior
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.