Design a verifier guided best of 8 strategy for a code generation product

Short answer·Medium·4.0 · 0·~3 min·Asked atHebbia Pinecone Tata Digital

Attempt it

Your code generation product already has a sandboxed unit test runner, and you can afford 8 samples per request. Design the test time strategy around the verifier, and defend why it beats both a single high effort chain and plain majority vote at this budget.

Free · 2 AI evals / day

The strategy has four steps. **Sample diverse candidates.** Generate 8 completions at moderate effort and a temperature that gives real diversity (something like 0.7-0.9 on most models). The goal is genuinely different approaches, not 8 near-identical strings. **Run all candidates through the verifier in parallel.** Feed each candidate into the sandboxed unit-test runner concurrently to keep total latency near the longest single run plus a little overhead, not 8x sequential. **Select the best passer.** If multiple candidates pass, pick the first-pass for latency-sensitive surfaces, or rank passing ones by style metrics (length, complexity, identifier naming) for quality-sensitive surfaces. **Handle the no-pass case.** If all 8 fail, do one repair round: feed each failing candidate plus its test error back to the model and ask for a revision. If still no passer, escalate to a higher-effort model or surface a graceful failure (return the best near-passer with a clear caveat). Why this beats plain majority vote: code outputs almost never match exactly, two correct implementations of the same function differ in whitespace, identifier naming, or structure. There is no majority to count. The verifier replaces the vote with direct correctness.

**Verifier-guided best-of-8 with parallel execution and a repair round.** Sample 8 candidates at moderate effort and temperature high enough for real approach diversity (0.7-0.9). Run all 8 through the sandboxed unit-test runner in parallel; total latency lands near the longest single run. Return the first passing candidate for latency-sensitive surfaces; for quality-sensitive surfaces, rank passers by code metrics (simplicity, length, identifier quality). If nothing passes, run one repair round: feed each failing candidate plus its specific test errors back for revision. If still no passer, escalate to a stronger model or surface a graceful failure with the best near-passer. **Why this beats majority vote.** Code outputs almost never match exactly. Two correct solutions differ in whitespace, names, and structure. Majority vote needs a counting key; freeform code has none with signal. Self-consistency works on closed-form outputs, not open-ended code. **Why it beats one long high-effort chain at matched compute.** 8 independent samples explore in parallel; one chain explores serially. A chain that commits to a wrong approach early burns its budget refining a doomed solution. Independent samples are robust to any single sample's bad start, and the verifier converts diversity into reliability. **Caveats.** Tests passing is not full correctness; coverage gaps slip. Run candidates concurrently and cache setup. Track no-pass rate per class as the signal for raising effort or changing models.

TL;DR

Sample 8 diverse candidates at moderate effort, run all through the sandbox in parallel, return the first passer; repair-round on failure; track no-pass rate.

Memory aid

Easy to grasp

Picture a hiring manager who has eight resumes for one role and a structured technical interview that takes thirty minutes. The smart play is not to read one resume in painstaking detail, nor to pick the most popular candidate by gut. The smart play is to give all eight the structured interview and hire the ones who pass. The interview is the verifier; it tells you exactly whether each candidate can do the job. For code generation, the verifier is your unit-test runner. Sample many candidates, let the tests judge them all, pick a passer. If none pass, ask them to revise based on the test failures and try once more. If still none pass, you know you need a tougher interviewer or a better candidate pool.

Key concepts

Reasoning Models

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Test-time compute strategies on reasoning models fall into a small number of canonical patterns: a single deep chain at high effort, k-sample majority vote, verifier-guided best-of-n, and tree of thought search. Which one to pick depends on whether you have a verifier available, how open-ended the output is, and how much compute you can afford per request.

A code-generation product with a sandboxed unit-test runner is the textbook case for verifier-guided best-of-n. The verifier is high-quality (test execution is a direct correctness signal), the output is open-ended (different code samples rarely match exactly), and the failure mode of single-chain reasoning, committing to a bad approach early, is exactly what diversity plus verification mitigates. With an 8-sample budget, the design has natural answers at every step.

Step 1: sampling for genuine diversity

The first move is to generate 8 candidates with enough diversity that the verifier has something to select among. Two settings matter.

Effort or thinking budget on a reasoning model should be moderate, not maximum. Per-sample budget matters less than per-sample independence at this strategy. Spending the full budget on 8 deep chains at high effort produces marginal accuracy gain per sample but burns 8x the cost; moderate effort gets most of the per-sample quality at substantially lower cost, and the verifier-selection step closes the gap.

Temperature should be high enough for genuine approach diversity. 0.7-0.9 is a common range, but the right value depends on the model: validate by sampling 8 candidates on a few representative prompts and computing code-similarity across them. If they collapse to near-duplicates (low edit distance, same control-flow shape), raise temperature; if they fragment into wildly different non-solutions, lower it.

Where the provider supports it, use different random seeds across the 8 samples; this guarantees independence even at lower temperatures. Without seed control, temperature is the only diversity lever and you have to lean on it harder.

Step 2: parallel sandbox execution

Step 3: the selection function

Step 4: the repair loop and the no-pass case

Why this beats the alternatives, and what to watch

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

# Verifier-guided best-of-8 for code generation
def best_of_n(prompt, n=8, model='reasoning', verifier=None):
    samples = [model.generate(prompt, temperature=0.7) for _ in range(n)]
    scored = []
    for s in samples:
        # Verifier: unit tests, type-check, lint, optional process reward
        ok, score = verifier.run(s.code)
        scored.append((score, s))
    scored.sort(key=lambda x: -x[0])
    return scored[0][1]  # highest verifier score sample

Real products, models, and research that use this idea.

Cursor and Windsurf agent modes use verifier-guided iteration on test runs to refine code edits in 2026.
OpenAI Codex-style benchmarks and LiveCodeBench evaluations show best-of-n with execution-based selection consistently outperforming single high-effort chains.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you measure whether 8 samples is the right n versus 4 or 16?

Track pass-rate as a function of n on an eval set; the curve typically saturates between 4 and 16 on code tasks. Pick n at the bend, where marginal pass-rate gain per sample becomes small.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Trying majority vote on code outputs. Two implementations of the same function rarely match character for character, so the vote has no signal to count.

60 second bullets to scan on the way to the call.

Describe a sampling strategy that produces real approach diversity
Explain why sandbox runs must be parallel for acceptable latency

Primary sources. Browse if you want the original framing.