Flashcard: what is self-consistency prompting and what does it produce as a final answer?

Flashcard·Easy·4.0 · 0·~30s·Asked atInfosys Wandb Weaviate·Relevant atAnthropic

Attempt it

Self-consistency is a prompting technique that samples N reasoning chains (typically 5 to 40) from the same chain-of-thought prompt at temperature > 0, then takes a majority vote across the final answers as the result. Plain CoT samples one reasoning chain (usually at temperature 0 or low temperature) and returns it directly. The intuition behind self-consistency is that hard reasoning problems often have many valid reasoning paths to the same correct answer, but only one or a few paths to any specific wrong answer. By sampling many paths and voting, the correct answer accumulates more votes than any wrong one. Introduced by Wang et al. in 2022, self-consistency boosted PaLM accuracy on GSM8K math word problems by roughly 17 percentage points over plain CoT. The cost is N times more output tokens, so it is reserved for hard problems where the accuracy gain justifies the multiplied cost. The final answer is just the most common one across the N samples; the reasoning chains are discarded after voting. Works best when answers are short and discrete (numbers, classifications, multiple-choice letters); does not work well for open-ended generation where outputs cannot be voted on.

Your answer

**Self-consistency** is a prompting technique that runs the same chain-of-thought prompt many times and picks the answer that appeared most often. The mechanics are simple. Start with a CoT prompt that asks the model to show its reasoning before giving the final answer. Instead of running it once at temperature 0 (which gives a single deterministic chain), run it N times (typically 5 to 40) at temperature > 0 (which introduces randomness). Each run produces a different reasoning chain and potentially a different final answer. Then take a majority vote across the final answers and return the most common one. The intuition is that for hard reasoning problems, there are usually many valid reasoning paths to the correct answer but only a few paths to any specific wrong answer. Different random seeds explore different paths. The correct answer accumulates more votes than any wrong one. It was introduced by Wang et al. in 2022 and boosted PaLM accuracy on GSM8K math problems by about 17 percentage points over plain CoT. The cost is N times more output tokens. So self-consistency is for hard problems where the extra accuracy is worth the multiplied cost, not for every prompt.

**Self-consistency** is a prompting technique that improves accuracy on hard reasoning tasks by sampling N chain-of-thought reasoning chains and majority-voting the final answer across them. The procedure has three steps. First, run a chain-of-thought prompt N times at temperature > 0 (typically 0.5 to 1.0) so each run produces a different reasoning chain. Second, extract the final answer from each chain. Third, count the answers and return the most common one. Typical N values are 5 to 40, with 40 being the upper end where the accuracy curve flattens. The intuition is that hard reasoning problems often have many valid paths to the correct answer but only a few paths to any specific wrong answer. Independent samples that converge on the same answer are signaling that answer is the consistent landing point of the model's reasoning. Wrong answers tend to be inconsistent across runs. The cost is real. N times more output tokens means N times the output bill, and output is already 3 to 5 times more expensive than input. Self-consistency is reserved for problems where the accuracy gain matters more than the cost: hard math, complex logical deduction, and high-stakes single-answer tasks. It is overkill for routine questions. The technique only works when answers are short and discrete enough to be voted on. Numbers, multiple-choice letters, classifications, and structured fields all work. Open-ended text generation does not; you cannot meaningfully majority-vote across paragraph-length answers. For those tasks, alternative techniques (LLM-as-judge ranking, best-of-N selection by reward model) replace majority voting.

TL;DR

Self-consistency samples N chain-of-thought reasoning chains at temperature > 0 from the same prompt, then majority-votes across the final answers; the most common answer wins.

Memory aid

Easy to grasp

Imagine asking 20 students to solve a tricky math problem and show their work. Each student takes a slightly different route to the answer because their thinking varies. Some get the wrong answer, but the wrong answers tend to be different from each other (one student forgot to carry the one, another misread the question). The correct answer shows up in many students' work because there are many valid routes to truth. If you count up the final answers and pick the most popular one, you usually pick the right one. Self-consistency does this with an LLM: it runs the same chain-of-thought prompt 20 times with some randomness, then picks the answer that came up most often. It is the wisdom of the crowd applied to one model.

Key concepts

Self-consistency Chain-of-thought Sampling

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Self-consistency is one of the most cited prompting techniques in the academic literature and one of the least deployed in production. The technique is conceptually elegant: many independent reasoning chains converge on the right answer; majority voting amplifies the signal. The accuracy gains on research benchmarks are real and substantial. But the cost overhead and the discrete-answer constraint make it a tool for specific use cases, not a general production pattern.

This deep dive walks through the mechanism, the intuition for why it works, the operational mechanics (sampling, voting, temperature), and the reasons it rarely shows up in production despite strong benchmark numbers.

The technique and the intuition

Self-consistency is a sampling-based extension of chain-of-thought prompting. The procedure has three steps. First, write a CoT prompt that asks the model to show its reasoning before giving the final answer. Second, run that prompt N times at temperature > 0 so each run produces a different reasoning chain. Third, extract the final answer from each chain and take the majority vote.

The intuition for why this works traces to the geometry of correct answers in language-model output space. For a hard reasoning problem, the correct answer is typically reached by many distinct valid reasoning paths. The model can decompose a math problem in several different ways, apply algebra at different points, choose different intermediate variables, and still arrive at the same final number. Wrong answers, by contrast, are typically reached by specific reasoning errors (forgot to carry the one, misread the question, applied the wrong formula). Different wrong reasoning errors lead to different wrong answers.

So when you sample many independent reasoning chains, the correct answer accumulates many votes (one per valid path) while any specific wrong answer accumulates only a few (one per reasoning error that happened to lead there). Majority voting picks the correct answer most of the time, even when each individual chain is correct only some of the time.

The 2022 paper showed PaLM-540B jumping from 56 percent on GSM8K with plain CoT to 74 percent with self-consistency at N=40. The technique is now a standard baseline in any reasoning-benchmark paper.

The mechanics: temperature, N, and aggregation

Where it works and where it does not

Why production deployment is rare

Variations and what they teach about ensembling

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Wang et al.'s 2022 self-consistency paper showed PaLM jumping from 56 percent to 74 percent on GSM8K math word problems by sampling 40 reasoning chains and majority-voting.
OpenAI's o1 and o3 reasoning models internally use sampling and voting style techniques in their hidden reasoning loop, though the exact mechanism is not public.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does temperature setting affect the accuracy gain from self-consistency?

Too low temperature (near 0) makes samples near-identical and voting meaningless; too high makes reasoning incoherent and final answers random. The sweet spot is usually 0.5 to 1.0, balancing diversity against coherence.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Using self-consistency on open-ended generation tasks; majority voting only works when answers are short and discrete enough to compare.

60 second bullets to scan on the way to the call.

What self-consistency is (sample N CoT chains, majority-vote the answer)
Why it works (many paths to right answers, few paths to specific wrong answers)

Primary sources. Browse if you want the original framing.