Flashcard: what is self-consistency prompting and what does it produce as a final answer? · Zenaique

Attempt it

Self-consistency is a prompting technique that samples N reasoning chains (typically 5 to 40) from the same chain-of-thought prompt at temperature > 0, then takes a majority vote across the final answers as the result. Plain CoT samples one reasoning chain (usually at temperature 0 or low temperature) and returns it directly. The intuition behind self-consistency is that hard reasoning problems often have many valid reasoning paths to the same correct answer, but only one or a few paths to any specific wrong answer. By sampling many paths and voting, the correct answer accumulates more votes than any wrong one. Introduced by Wang et al. in 2022, self-consistency boosted PaLM accuracy on GSM8K math word problems by roughly 17 percentage points over plain CoT. The cost is N times more output tokens, so it is reserved for hard problems where the accuracy gain justifies the multiplied cost. The final answer is just the most common one across the N samples; the reasoning chains are discarded after voting. Works best when answers are short and discrete (numbers, classifications, multiple-choice letters); does not work well for open-ended generation where outputs cannot be voted on.

Your answer

**Self-consistency** is a prompting technique that runs the same chain-of-thought prompt many times and picks the answer that appeared most often. The mechanics are simple. Start with a CoT prompt that asks the model to show its reasoning before giving the final answer. Instead of running it once at temperature 0 (which gives a single deterministic chain), run it N times (typically 5 to 40) at temperature > 0 (which introduces randomness). Each run produces a different reasoning chain and potentially a different final answer. Then take a majority vote across the final answers and return the most common one. The intuition is that for hard reasoning problems, there are usually many valid reasoning paths to the correct answer but only a few paths to any specific wrong answer. Different random seeds explore different paths. The correct answer accumulates more votes than any wrong one. It was introduced by Wang et al. in 2022 and boosted PaLM accuracy on GSM8K math problems by about 17 percentage points over plain CoT. The cost is N times more output tokens. So self-consistency is for hard problems where the extra accuracy is worth the multiplied cost, not for every prompt.

**Self-consistency** is a prompting technique that improves accuracy on hard reasoning tasks by sampling N chain-of-thought reasoning chains and majority-voting the final answer across them. The procedure has three steps. First, run a chain-of-thought prompt N times at temperature > 0 (typically 0.5 to 1.0) so each run produces a different reasoning chain. Second, extract the final answer from each chain. Third, count the answers and return the most common one. Typical N values are 5 to 40, with 40 being the upper end where the accuracy curve flattens. The intuition is that hard reasoning problems often have many valid paths to the correct answer but only a few paths to any specific wrong answer. Independent samples that converge on the same answer are signaling that answer is the consistent landing point of the model's reasoning. Wrong answers tend to be inconsistent across runs. The cost is real. N times more output tokens means N times the output bill, and output is already 3 to 5 times more expensive than input. Self-consistency is reserved for problems where the accuracy gain matters more than the cost: hard math, complex logical deduction, and high-stakes single-answer tasks. It is overkill for routine questions. The technique only works when answers are short and discrete enough to be voted on. Numbers, multiple-choice letters, classifications, and structured fields all work. Open-ended text generation does not; you cannot meaningfully majority-vote across paragraph-length answers. For those tasks, alternative techniques (LLM-as-judge ranking, best-of-N selection by reward model) replace majority voting.

Quick reference

TL;DR

Self-consistency samples N chain-of-thought reasoning chains at temperature > 0 from the same prompt, then majority-votes across the final answers; the most common answer wins.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine asking 20 students to solve a tricky math problem and show their work. Each student takes a slightly different route to the answer because their thinking varies. Some get the wrong answer, but the wrong answers tend to be different from each other (one student forgot to carry the one, another misread the question). The correct answer shows up in many students' work because there are many valid routes to truth. If you count up the final answers and pick the most popular one, you usually pick the right one. Self-consistency does this with an LLM: it runs the same chain-of-thought prompt 20 times with some randomness, then picks the answer that came up most often. It is the wisdom of the crowd applied to one model.

Key concepts

Self-consistency Chain-of-thought Sampling

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

6 min: define self-consistency as sample and vote on CoT, explain the many paths to right answers intuition, name temperature requirement, walk through cost trade, name when it does not apply, explain why production deployment is rare.

Real-world examples

Real products, models, and research that use this idea.

Wang et al.'s 2022 self-consistency paper showed PaLM jumping from 56 percent to 74 percent on GSM8K math word problems by sampling 40 reasoning chains and majority-voting.
OpenAI's o1 and o3 reasoning models internally use sampling and voting style techniques in their hidden reasoning loop, though the exact mechanism is not public.
Many research benchmark pipelines (MATH, AIME, GPQA) report self-consistency scores at N=8 or N=32 as a standard reporting metric for comparing models.
Production teams rarely deploy self-consistency directly; the N times cost overhead usually loses to a frontier reasoning model or to fine-tuning on the target task.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does temperature setting affect the accuracy gain from self-consistency?

Sign in or go Pro to see the approach for this follow-up.

QWhat is the difference between self-consistency and best-of-N with a reward model?

Sign in or go Pro to see the approach for this follow-up.

QWhy is self-consistency rarely deployed in production despite strong research benchmarks?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Using self-consistency on open-ended generation tasks; majority voting only works when answers are short and discrete enough to compare.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.