Spot the issue with this production prompt template that adds Chain-of-Thought to every customer-support query, including simple fact-lookup ones.

The system prompt tells the model to follow a step by step process for every user question. That is universal chain-of-thought, and on a customer-support workload it is the wrong default. Look at the example. The user asks what time the store closes. That is a one-step fact lookup. The assistant's thinking block walks through 'consider what day it is, look up hours, check for holiday hours, format the response' before giving the answer. Every one of those internal tokens costs money and latency, and the answer would have been just as good without them. The fix is conditional CoT. Apply step by step reasoning when the question actually has multiple steps (refund calculations, troubleshooting flows, multi-fact lookups). Skip it for greetings, simple facts, and one-shot acknowledgments. A small classifier or even a keyword heuristic can route the request to the right prompt template.

The error is universal CoT. The system prompt instructs the model to walk a step by step process for every user question, which is wrong when most customer-support traffic is single-step. The example shows the symptom cleanly: the user asks store hours, and the assistant produces a thinking block that walks through irrelevant subproblems before giving a one-line answer. The production cost is real on two axes. First, every CoT round adds output tokens, typically 60 to 150 of them, on traffic that does not benefit. At ten thousand to a hundred thousand queries per day, that is a multiple on the cost line. Second, the latency hit pushes simple routes past their p95 budget, which is exactly where customer-support traffic is most sensitive. The quality risk is the subtler one. On simple lookups, the model with CoT sometimes invents a reasoning chain that drifts off the correct answer. The thinking block here speculates about holiday hours that may or may not apply. That kind of drift turns a clean answer into a hedged or wrong one. The fix is conditional CoT. A cheap classifier or keyword rule decides whether the request gets the step by step scaffold. Simple lookups get a direct-answer prompt; multi-step requests get the CoT prompt.