Explain why self-consistency requires temperature > 0 and quantify the gain it produces over single-pass CoT on math benchmarks. · Zenaique

Attempt it

A teammate is implementing self-consistency: 'I'll sample N=10 CoT reasoning paths and vote on the final answer. I'm running temperature=0 for reproducibility.' Walk them through why this doesn't work, what temperature to use instead, and the empirical gain they should expect on GSM8K-style math benchmarks. Be specific about the mechanism and the numbers.

Free · 2 AI evals / day

The plan as written cannot improve over single-pass CoT. At temperature zero the language model is deterministic: given the same prompt, every run produces exactly the same tokens. Ten identical samples voted on by majority is one sample's outcome. The N=10 cost is paid; the voting benefit is zero. The fix is to raise the temperature into the range where reasoning paths actually diverge. The standard pick is temperature 0.7. At that level the model still produces high-quality reasoning, but each sample explores a slightly different path. The vote then concentrates on the answer that the most reasoning paths agree on. The gain you should expect on math benchmarks is significant. The original Wang et al. paper showed 17.9 points on GSM8K with PaLM at N=40. Reproductions on modern frontier models show 10 to 20 points on GSM8K-class reasoning benchmarks. In production, N=5 to N=10 at temperature 0.7 captures most of the lift at much lower cost than N=40. Past N=10 the marginal gain flattens fast. The reproducibility tension is real. Temperature zero is great for deterministic eval. It is incompatible with self-consistency. You have to pick one for any given route.

**Self-consistency** requires temperature greater than zero because the mechanism is voting over diverse reasoning paths, and temperature zero produces no diversity. At T=0 the decoder always picks the argmax token; N runs produce N identical sequences; voting reduces to a single sample's outcome. The majority-vote formula is straightforward: $$\text{answer} = \arg\max_y \sum_{i=1}^N \mathbb{1}[y_i = y]$$ With N independent CoT samples, each yields its own final answer `y_i`. The vote concentrates on the answer that appears most often. The asymmetry it exploits: correct answers tend to be reached via multiple reasoning paths while incorrect answers scatter. The vote picks up that signal. **Temperature recommendation.** Use 0.7 as the production default. The acceptable band is 0.5 to 0.9. Below 0.5 the diversity is too low for the vote to extract signal. Above 0.9 the reasoning quality starts to degrade. **Empirical gains.** Wang et al. 2022 reported +17.9 points on GSM8K with PaLM-540B at N=40 (56.5 to 74.4 percent). SVAMP gained 11 points; AQuA-RAT gained 12. Modern reproductions on GPT-4 class and Claude class frontier models show 10 to 20 absolute points on multi-step reasoning benchmarks at N=5 to N=10. Cost: N times the single-sample inference. The diminishing-returns curve flattens past N=10. The right production target is N=5 to N=10 unless the task is high-value enough to justify more. Now the reproducibility tension. Temperature zero is valuable for deterministic eval, where you want bit-identical output across runs. It is incompatible with self-consistency, which requires the very stochasticity reproducibility forbids. The route either accepts non-determinism for the self-consistency lift or runs single-pass CoT at T=0 for deterministic eval. You cannot have both on the same call.

Quick reference

TL;DR

Temperature zero produces identical samples, so the vote does nothing; switch to T around 0.7 with N=5 to 10 and expect 10 to 20 points on GSM8K-class benchmarks.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine you ask the same friend to solve a hard puzzle ten times, but you tell them to give the exact same answer every time. You will get ten identical answers. Voting on ten identical answers tells you nothing new. That is what temperature zero does to a language model. Now tell the friend to think about the puzzle freshly each time, allowing their reasoning to wander a little. You will get a mix of answers, with the correct one usually winning the count. Self-consistency only works in the second mode. Forcing it into the first mode by running at temperature zero gives you all the cost of ten attempts with the answer quality of one.

Key concepts

Prompt Engineering Agents

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: deterministic decoding kills voting, temperature 0.7 with N=5 to 10 captures most of the lift, GSM8K gain anchors at 10 to 20 points, reproducibility and amplification are two different routes.

Real-world examples

Real products, models, and research that use this idea.

Math-tutoring routes in production agentic apps run N=5 to N=10 self-consistency at temperature 0.7 to catch arithmetic slips that single-pass CoT misses.
Anthropic Claude Opus 4.7 with extended thinking and OpenAI o-series models internalize part of the self-consistency mechanism via sampling inside the reasoning trace.
Eval pipelines for GSM8K and MATH-class benchmarks use self-consistency at N=10 as a strong baseline before reporting any single-pass numbers.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you adapt self-consistency for tasks without a clean discrete answer?

Sign in or go Pro to see the approach for this follow-up.

QHow does seeded sampling change the reproducibility picture?

Sign in or go Pro to see the approach for this follow-up.

QWhere do modern reasoning models reduce the marginal value of external self-consistency?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Setting temperature=0 for reproducibility and then wondering why N=10 self-consistency gives identical accuracy to N=1 at ten times the cost.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.