Diagnose why majority voting stopped working on free form summary outputs

**Majority voting works on math because the final answer is a single discrete token.** Five samples of a hard math problem can all produce the same final number, even if the reasoning chains that got there are different. Tallying the final answers and picking the most common is straightforward. **Summaries are free-form prose.** Five samples of the same document produce five different strings, even at moderate temperature. Exact-match tallying treats each string as its own unique answer; you get five singleton votes and no majority. The voting mechanism is structurally incompatible with the output type. The fix is to change the selection mechanism. Instead of counting votes, you score candidates. A few options work in practice. An LLM-as-judge scores each summary against a rubric (faithfulness, conciseness, coverage); you pick the highest-scoring summary. A reward model trained on summary preferences predicts a quality score per candidate. Task-specific checks (ROUGE-overlap with source, factuality check against retrieved chunks) can also serve as scorers. This is called best-of-n by score rather than by count. The other options are wrong. Temperature does not break voting statistics; it just makes outputs more diverse, which is desired for self-consistency. Majority voting works fine for odd or even sample counts. Reasoning models can produce excellent long-form prose; the issue is the selection mechanism, not the generation capability.

**The structural mismatch is exact-match counting versus continuous-output space.** Self-consistency, as introduced by Wang et al. 2022, works by sampling multiple reasoning chains, extracting the final answer from each, and tallying the most common final answer. The mechanism depends on the final answer being a small discrete token: a number, a multiple-choice letter, a yes/no, a short answer span. Five chains can independently arrive at '42' and the count is meaningful. Summaries break this assumption at the answer extraction step. There is no compact final answer to extract; the entire output is the answer. Two summaries of the same document at temperature 0.7 produce different word choices, different sentence orderings, different emphasis. Exact-match tallying gives five singletons, and the most common answer mechanism returns an arbitrary first-occurrence with count 1. The principled fix is to replace counting with scoring. Best-of-n with a scoring function works for any output type, including free-form prose. The scoring function can be: - LLM-as-judge: a separate model evaluates each candidate against a rubric and returns a numeric score. - Reward model: a model trained on preference data predicts a quality score. - Task-specific verifier: faithfulness check against source, length constraints, presence of required entities. The production pattern is to sample n candidates, score each, and pick the top-scoring one. This generalizes self-consistency from discrete-answer tasks to any task where you can define a scoring function. The wrong options each represent a misdiagnosis. Temperature is a red herring; higher temperature is what self-consistency needs for sample diversity. Sample-count parity is irrelevant. Reasoning models produce excellent prose; the bottleneck is the selection step.