Which candidate does best of N sampling ship given these reward scores?
Best of 4 sampling is enabled at deployment. For one prompt, the policy samples four candidates and the reward model scores them: A = 1.3, B = 2.1, C = 0.4, D = 1.9. Which candidate does best of N return to the user?
Candidate B at 2.1 ships. Best-of-N is a pure argmax over reward-model scores; no averaging, no weights, no training update.
Imagine a baker pulls four cookies from the oven and a taste-tester rates each one out of ten. The baker is not blending them or averaging them. The baker picks the cookie the tester rated highest and serves that one to the customer. The other three get thrown away for that order. The next customer gets four new cookies and the same drill. The taste-tester is the reward model and the baker is the best-of-N policy. Picking the top scorer is the whole strategy. The other candidates were generated only to give the tester options to choose from.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Best-of-N is the simplest production use of a reward model after training and one of the most useful inference-time knobs in modern LLM serving. The mechanism is plain: generate N responses, score each with the RM, return the top scorer.
For the scores A=1.3, B=2.1, C=0.4, D=1.9, B at 2.1 is the maximum, so B ships. The other three responses are discarded after this prompt and play no role in future requests.
The exact mechanism
For a given prompt x, the policy temperature-samples N independent candidate responses y_1, y_2, ..., y_N. The reward model evaluates each candidate independently, producing scores r(x, y_1), r(x, y_2), ..., r(x, y_N). The system returns the single candidate with the highest score:
Here, the scores are 1.3, 2.1, 0.4, 1.9. The maximum is 2.1, attached to candidate B, so B is the response sent to the user. A, C, and D are discarded.
A few things do not happen. The scores are not averaged. The candidates are not blended into a hybrid response. There is no temperature schedule that weights candidates by their RM score; it is a hard argmax. And the policy weights are unchanged after this prompt. The next prompt will see exactly the same policy, generate four fresh candidates, and the RM will judge those independently. BoN is stateless across requests.
The size of the gain on this prompt is easy to read off. Serving one of the four candidates at random would score on average. BoN returns 2.1, a lift of about 0.68 reward units for four generations plus four RM calls. The drift from the base policy is small. The KL between the BoN output distribution and the base policy is at most nats, which for N = 4 is about 0.64. Gao, Schulman and Hilton used this formula citing Stiennon et al. 2020. Beirami et al. 2024 proved that it is an upper bound, not an exact value.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- WebGPT's best model was GPT-3 fine-tuned by behavior cloning, with rejection sampling (best-of-n) against a reward model trained to predict human preferences
- Llama 3 post-training sampled K outputs per prompt (typically 10 to 30) and used its reward model to keep the best candidate for SFT data
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does BoN quality scale with N, and where does it stop helping?
Discuss the BoN-KL equivalence, the empirical curve plateau, and the Goodhart peak past which gold quality declines despite rising RM scores.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Averaging the four scores, blending the responses, or thinking BoN is a training step rather than an inference-time argmax over a learned reward model.
60 second bullets to scan on the way to the call.
What argmax means in the BoN context
Whether BoN updates policy weights (no)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.