Predict which candidate best of N sampling ships given these reward scores

Attempt it

Best of 4 sampling is enabled at deployment. For one prompt, the policy samples four candidates and the reward model scores them: A = 1.3, B = 2.1, C = 0.4, D = 1.9. Which candidate does best of N return to the user?

TL;DR

Candidate B at 2.1 ships. Best-of-N is a pure argmax over reward-model scores; no averaging, no weights, no training update.

Memory aid

Sign in to see the mnemonic that makes this stick.

Easy to grasp

Imagine a baker pulls four cookies from the oven and a taste-tester rates each one out of ten. The baker is not blending them or averaging them. The baker just picks the cookie the tester rated highest and serves that one to the customer. The other three get thrown away for that order. Next customer, four new cookies, same drill. The taste-tester is the reward model and the baker is the best-of-N policy. Picking the top scorer is the whole strategy. The other candidates were generated only to give the tester options to choose from.

Key concepts

RLHF

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Best-of-N is the simplest production use of a reward model after training and one of the most useful inference-time knobs in modern LLM serving. The mechanism is plain: generate N responses, score each with the RM, return the top scorer. The depth comes from understanding what BoN does to the effective output distribution, where it stops helping, and why it is often the right first choice for a small team.

For the specific scores in this question — A=1.3, B=2.1, C=0.4, D=1.9 — the answer is unambiguous: B at 2.1 is the maximum, so B ships. The other three responses are discarded after this prompt and play no role in future requests.

This deep dive covers the precise mechanism, the connection between BoN and KL-constrained RL, the cost and overoptimization tradeoffs, and the operational reasons BoN is often deployed before any RL training.

The exact mechanism

For a given prompt x, the policy temperature-samples N independent candidate responses y_1, y_2, ..., y_N. The reward model evaluates each candidate independently, producing scores r(x, y_1), r(x, y_2), ..., r(x, y_N). The system returns the single candidate with the highest score:

y^* = \arg\max_{i} r(x, y_i)

For this question, the scores are 1.3, 2.1, 0.4, 1.9. The maximum is 2.1, attached to candidate B, so B is the response sent to the user. A, C, and D are discarded.

A few things are pointedly not happening. The scores are not averaged. The candidates are not blended into a hybrid response. There is no temperature schedule that weights candidates by their RM score; it is a hard argmax. And critically, the policy weights are unchanged after this prompt. The next prompt will see exactly the same policy, generate four fresh candidates, and the RM will judge those independently. BoN is stateless across requests.

Walk through a concrete number example. Say N = 8 and reward scores are $[2.1, 1.4, 3.2, 0.9, 2.8, 1.7, 2.5, 1.1]$ . The argmax is index 2, score 3.2. The base policy with temperature 0.7 produced those eight candidates with roughly uniform probability after reward-weighted decoding. The expected reward under uniform sampling is $\bar{r} = 1.96$ . BoN's expected reward is $E[\max] \approx 3.0$ . The lift is 1.04 reward units for 8x inference cost. The implicit KL paid for that lift is, in the Gaussian-tail approximation of Stiennon et al. 2020 and Hilton 2023, roughly $\log N - \frac{N-1}{N}$ nats, here about 1.2.

Why BoN works — the implicit policy improvement

Cost and where BoN stops being the right tool

When teams reach for BoN first

Sign in to unlock the full deep dive.

Situations where this technique stops working.

Sign in to see when this approach fails.

2–4 min · Everything important, quickly.

Sign in to see the quick scan of the deep dive.

Real products, models, and research that use this idea.

OpenAI's WebGPT and early ChatGPT-era systems used BoN re-ranking with a learned reward model at inference time before they had full RL infrastructure
Cohere's Rerank API operationalizes the same argmax over candidates pattern for retrieval and generation re-ranking

Sign in to see more production examples.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does BoN quality scale with N, and where does it stop helping?

A

Discuss the BoN-KL equivalence, the empirical curve plateau, and the Goodhart peak past which gold quality declines despite rising RM scores.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Averaging the four scores, blending the responses, or thinking BoN is a training step rather than an inference-time argmax over a learned reward model.

Sign in to see all red flags and common mistakes.

60 second bullets to scan on the way to the call.

What argmax means in the BoN context
Whether BoN updates policy weights (no)

Sign in to unlock the revision sheet.