Chain-of-thought helps when the task actually needs intermediate steps to track; on single-step recall, translation, or greetings it just burns tokens.
Imagine someone asks what is two plus two. You answer four, instantly. Now imagine they ask: a baker has twelve cupcakes, sells half, then bakes ten more, how many does she have. You probably mutter the steps out loud or sketch them. That muttering is what chain-of-thought does for an LLM. It only earns its keep when there are real intermediate steps. Asking the model to think step by step about a single fact lookup just adds typing without making the answer any better, and sometimes makes it worse by sneaking in fake reasoning.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Chain-of-thought is the most over-recommended pattern in prompt engineering. The 2022 papers showed real, large gains on real, hard tasks. The interview-circuit version of the claim is that CoT always helps. The production version is that CoT is a real lever on a specific shape of task and a tax everywhere else.
This deep dive walks through the mechanism (why CoT helps when it does), the empirical boundary (where it stops helping), the failure modes (where it hurts), and the modern model-layer evolution (reasoning models that internalize the pattern). The goal is a working heuristic an engineer can apply per route, not a vibe.
The punchline is task structure, not task difficulty. A two-step task benefits from CoT. A one-step task does not, no matter how hard the one step is.
The mechanism: why CoT helps on multi-step tasks
Language models generate one token at a time. Each token is produced from the model's current state, which depends on everything in the prompt up to that point. When a task has multiple subproblems (intermediate values, partial results, sub-conclusions), producing each subproblem as explicit tokens gives the model a checkpoint to anchor the next subproblem on.
Without CoT, the model has to collapse the whole computation into one guess at the answer position. With CoT, each subproblem becomes part of the input the model sees while producing the next subproblem. That changes the conditional distribution from P(final | prompt) to P(final | prompt, step1, step2, ...), which is exactly the decomposition the math is set up for.
This is also why CoT is most reliable on tasks that genuinely decompose: arithmetic word problems (carry intermediate values), multi-hop QA (chain facts), planning (track state), and code generation with intermediate computations.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI's o-series reasoning models bake CoT into the model itself, with the API exposing an explicit reasoning mode for tasks that need it.
- Anthropic Claude Opus 4.7 offers extended thinking that the developer toggles per request, mirroring conditional-CoT logic at the model layer.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhen does CoT actually hurt accuracy on a task?
Simple retrieval tasks where the model invents a reasoning chain that drifts; cite published evidence and the latency cost on hot paths.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Believing chain of thought is a universal accuracy booster and adding 'think step by step' to every prompt, including single-step tasks where it just adds latency and sometimes invents reasoning that misleads the answer.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.