Same topic, related formats. Practice these next.
RLOO generates k rollouts per prompt and uses the mean of the other k-1 as each rollout's baseline. No critic, no leakage of a rollout into its own baseline, unbiased advantages.
Imagine four students each solve the same math problem, and a teacher scores all four. To decide how good Alice's solution is compared to the rest, you average the scores of the other three students and compare Alice to that average. You do not include Alice's own score in the average, because that would bias the comparison toward her. Then you do the same for Bob, comparing him to the average of the other three. That tiny statistical trick is the whole heart of RLOO. The model generates several attempts at the same prompt, scores them, and learns more from the attempts that beat the typical attempt and less from the ones that lagged behind.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: parse the acronym, state the leave-one-out formula, explain why excluding the own reward keeps the baseline unbiased, contrast with PPO (critic) and best-of-k (group max), then sketch the k-versus-cost tradeoff.
| Method | Baseline source | Critic network? | Infrastructure complexity |
|---|---|---|---|
| PPO (actor-critic) | Learned value function | Yes (trained jointly) | High; value head, clipping, GAE |
| RLOO | Mean of other k-1 rollouts in same group | No | Low; just rollouts + REINFORCE |
| Vanilla REINFORCE | Running average or none | No | Lowest but highest variance |
| RAFT / expert iteration | Best-of-k filtering, supervised on top | No | Low; not a policy gradient |
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Including rollout i's own reward in the baseline (the group mean of all k), which biases the advantage toward zero and reduces signal. Leave-One-Out excludes the rollout's own score by definition.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.