Compare best-of-N at inference against RL training for a small product team

**Best-of-N (BoN)** and RL training are two ways to use the same reward model, with very different tradeoffs for a small team. BoN ships fast. Generate N candidates per prompt, score them with the reward model, return the top one. No training infrastructure, no PPO loop, no KL tuning. The policy weights stay exactly as they were, so if BoN ever causes a problem you can turn it off and you are back to the original behavior. Quality lift for moderate N (4-16) is often comparable to a mild RL run against the same RM. The cost is at serving. Every request now pays for N generations plus N reward-model evaluations, so latency rises and inference cost grows linearly in N. And pushing N very high starts overoptimizing the reward model, same Goodhart pressure RL hits, just expressed through search. RL training is the opposite shape. You pay a one-time cost to bake quality into the weights, then serve a single cheap sample per request. The lift can exceed BoN's, and the behavior persists. But you need real infrastructure, careful KL management, and the ability to debug reward hacking. For a five-person team with no RL infra, start with BoN.

**Default recommendation:** start with best-of-N, plan to graduate to RL training when traffic or latency forces it. BoN mechanics: sample N candidates, score each with the RM, return the argmax. The policy is untouched, so rollback is free, A/B testing is clean, and the infrastructure footprint is a single inference-time wrapper. The team can ship this in days. For N between 4 and 16, the quality lift on standard chat tasks is often comparable to a well-tuned mild RL run against the same RM. BoN costs accumulate at serving time. Every request pays N generations and N RM scoring calls, latency climbs, and large N starts overoptimizing the RM at inference: the same Goodhart pressure RL training faces, expressed through search rather than gradient descent. Most production deployments cap N at 4-16. RL training (PPO, GRPO, or iterative DPO) amortizes the optimization into a one-time training run. Serving becomes a single sample per request. The quality ceiling is higher than BoN's because the policy can learn behaviors BoN can only stumble onto. The cost is infrastructure: distributed training, KL controllers, reward-hacking monitoring, rollback discipline. **Crossover rules.** Use BoN when traffic is modest, latency budgets accommodate N samples, iteration speed matters, or rollback safety is critical. Move to RL when serving cost (N times inference times traffic) outgrows the one-time training cost, when latency cannot absorb N samples, or when the improvement needs to persist in the weights for distillation, offline use, or downstream fine-tunes.