Click any words you think contain an error. Click again to unmark.
Mark at least one word to submit.
The system prompt tells the model to follow a step by step process for every user question. That is universal chain-of-thought, and on a customer-support workload it is the wrong default.
Look at the example. The user asks what time the store closes. That is a one-step fact lookup. The assistant's thinking block walks through 'consider what day it is, look up hours, check for holiday hours, format the response' before giving the answer. Every one of those internal tokens costs money and latency, and the answer would have been just as good without them.
The fix is conditional CoT. Apply step by step reasoning when the question actually has multiple steps (refund calculations, troubleshooting flows, multi-fact lookups). Skip it for greetings, simple facts, and one-shot acknowledgments. A small classifier or even a keyword heuristic can route the request to the right prompt template.
The error is universal CoT. The system prompt instructs the model to walk a step by step process for every user question, which is wrong when most customer-support traffic is single-step. The example shows the symptom cleanly: the user asks store hours, and the assistant produces a thinking block that walks through irrelevant subproblems before giving a one-line answer.
The production cost is real on two axes. First, every CoT round adds output tokens, typically 60 to 150 of them, on traffic that does not benefit. At ten thousand to a hundred thousand queries per day, that is a multiple on the cost line. Second, the latency hit pushes simple routes past their p95 budget, which is exactly where customer-support traffic is most sensitive.
The quality risk is the subtler one. On simple lookups, the model with CoT sometimes invents a reasoning chain that drifts off the correct answer. The thinking block here speculates about holiday hours that may or may not apply. That kind of drift turns a clean answer into a hedged or wrong one.
The fix is conditional CoT. A cheap classifier or keyword rule decides whether the request gets the step by step scaffold. Simple lookups get a direct-answer prompt; multi-step requests get the CoT prompt.
Quick reference
TL;DR
Forcing chain-of-thought on every customer-support query, including single-step lookups, burns tokens and latency on traffic that does not benefit, and sometimes invents misleading reasoning.
Imagine a help desk where every clerk has to explain their thinking step by step before answering, even when the question is what time does the store close. The clerk takes longer, costs more, and sometimes talks themselves into the wrong answer because they over-think a simple lookup. The fix is to let the clerk answer simple questions directly and only walk through their reasoning when the question actually needs steps, like a refund calculation or a troubleshooting flow. The same rule applies to LLM prompts. Forcing think step by step on every query is the bug.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Workloads where every request is genuinely multi-step (eligibility checks, refund calculations) Conditional routing has nothing to route away from; CoT belongs on the default prompt.
Internal tools used by trained operators where latency is irrelevant The cost and UX downsides disappear, so the routing layer is overhead with no payoff.
When you already deploy a reasoning model with thinking mode The conditional decision moves to the API toggle; manual CoT in the prompt is redundant and can collide with the model's internal trace.
Quick scan · 2–4 min
Everything important, quickly.
3 min: universal vs conditional CoT + cost and latency lines on support traffic + the invented-reasoning failure mode + production classifier pattern.
Real-world examples
Real products, models, and research that use this idea.
1OpenAI Structured Outputs lets you ship a direct-answer schema for simple support queries while reserving the reasoning-mode endpoint for harder paths.
2Anthropic Claude Opus 4.7 exposes extended thinking as a per-request toggle, so support teams can enable it only on routes that classify as multi-step.
3Zendesk and Intercom AI assistants use lightweight intent classifiers to pick between direct-answer and reasoning prompts, mirroring conditional CoT routing in production.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you classify customer-support queries into CoT vs direct routes?
Sign in or go Pro to see the approach for this follow-up.
QWhat happens if you A/B universal CoT against conditional CoT on this workload?
Sign in or go Pro to see the approach for this follow-up.
QDoes extended thinking mode change the answer here?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Hard-coding 'think step by step' into the system prompt for every customer-support query, ignoring that fact-lookup traffic dominates and CoT adds latency and cost without quality gain on that majority.