Tree of Thoughts (Yao et al. 2023) explores multiple reasoning branches at each step. On what kind of problem does the tree structure pay off over a single-thread Chain of Thought?
**Tree of Thoughts** (often shortened to ToT) is a prompting pattern where the model explores several possible next reasoning steps from each state, evaluates them, prunes the bad ones, and continues searching down the promising branches. The result is a tree of partial reasoning paths rather than a single linear chain.
The payoff comes from being able to detect a wrong path before reaching the final answer. If you can tell partway through that a branch is going nowhere, you save the rest of the work on that branch and put compute into branches that look better. Classic ToT examples are Game of 24, crossword puzzles, and small-scale planning problems, where partial states are easy to score.
For problems where every path looks equally good until the very end, ToT does not help. The pruning step has nothing to act on, so you just paid 5 to 50 times more compute to end up at the same answer as a single CoT chain. The pattern is targeted, not universal.
ToT pays off when the problem has two specific properties at once. First, multiple plausible reasoning paths exist from each intermediate state, so there is something to branch over. Second, partial paths can be evaluated before reaching the final answer, so there is a signal to prune on.
This is the classic profile of a search problem. Game of 24, crosswords, small-scale planning, and constraint-satisfaction tasks all fit. The model proposes K candidate next steps from each state, a value function (often another prompt) scores each partial path, and the search keeps the top-B branches and continues. The structure is a beam search over reasoning.
Compare with Chain of Thought, which is one linear chain. CoT is more efficient when the problem is well-conditioned and a competent first attempt usually succeeds. Compare with self-consistency, which samples N parallel chains and votes on their final answers. Self-consistency assumes parallel independence; ToT assumes intermediate evaluability.
The production caveat is cost. ToT typically runs at 5 to 50 times the base inference cost depending on branching factor K and tree depth D. That is fine for high-value puzzle-style tasks. It is wasteful for tasks where partial reasoning cannot be scored or where a single CoT chain solves the problem most of the time.
Quick reference
TL;DR
ToT pays off when the problem has multiple plausible reasoning paths AND wrong paths can be evaluated and pruned before the final answer: puzzle-solving, planning, constrained search.
Imagine solving a maze. Chain of Thought is like committing to one direction at every fork and only finding out you were wrong when you hit a dead end. Tree of Thoughts is like trying a few directions at each fork, checking which ones look promising, and abandoning the bad ones early. The tree pays off when you can tell partway through that a path is going nowhere. If every path looks the same until the end, the extra exploration is just wasted effort.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tasks where partial reasoning paths cannot be evaluated reliably Without a pruning signal, the top-B selection at each depth is essentially random, so the extra branching just multiplies cost without benefit.
Latency-sensitive user-facing requests ToT runs at 5 to 50 times base inference cost; a tree-deep search blows past any sub-second SLA on a typical chat surface.
Tasks where a single CoT chain succeeds most of the time If the base success rate is high, the marginal gain from search is small and the cost multiplier rarely pays off.
When a learned verifier is available and you can do best-of-N instead Verifier-guided best-of-N is often cheaper and stronger than ToT when only the final answer is scoreable.
Quick scan · 2–4 min
Everything important, quickly.
4 min: ToT as beam search + branching cost K * B * D + intermediate-evaluation requirement + contrast with CoT and self-consistency + when to skip.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's research on agentic planning uses tree-style search over candidate tool calls, evaluated by Claude Opus 4.7 acting as the scorer.
2DSPy's BootstrapWithSearch wraps a tree-structured optimizer that prunes prompt candidates by held-out eval performance.
3Game of 24 and small-scale crossword solvers in the original Yao et al. paper remain the canonical ToT benchmarks.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you implement the value function that scores partial reasoning paths?
Sign in or go Pro to see the approach for this follow-up.
QWhen would you pick ToT over verifier-guided best-of-N?
Sign in or go Pro to see the approach for this follow-up.
QHow do you bound ToT's cost in production?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reaching for ToT on tasks where partial reasoning paths cannot be evaluated, so the pruning step has no signal and the tree just multiplies cost without benefit.