What does Leave One Out in RLOO compute?

MCQ·Medium·4.0 · 0·~1 min·Asked atAirbnb OpenAI Voyage Ai·Relevant atAnthropic Cohere Fireworks Ai Google

Attempt it

The right answer is A. RLOO stands for REINFORCE Leave-One-Out, and the baseline is literally what the name says. The setup: for each training prompt, the model generates k different responses (rollouts), and a reward model scores each one. Each rollout then needs an advantage: how much better or worse it scored than a typical response to the same prompt. The baseline answers what 'typical' means. RLOO computes it as the average reward of the other k-1 rollouts, not including the rollout you are scoring. If rollout 1 scored 0.8 and the other three scored 0.5, 0.6, and 0.7, its baseline is their mean, 0.6. Its advantage is 0.8 minus 0.6, or 0.2. The leave-one-out construction matters because it keeps the baseline unbiased. If you included rollout 1's own reward in the average, the comparison would partially compare rollout 1 to itself. That artificially shrinks the advantage and weakens the gradient signal. The key win over PPO is that RLOO needs no separate critic network, so option B is wrong. C uses a running average over the whole run, which lags as the policy improves. D uses the group max, so every advantage is zero or negative.

**Option A** is correct. RLOO (REINFORCE Leave-One-Out) computes the baseline for each rollout i as the mean reward of the other k-1 rollouts in the same group. Here is the full setup. For each prompt, generate k rollouts. Score each rollout with the reward model. For rollout i, compute the advantage: $$A_i = r_i - \frac{1}{k-1} \sum_{j \neq i} r_j$$ The sum excludes rollout i, which is the leave-one-out construction. This is what keeps the advantage unbiased. Rollout i's reward is not part of its own baseline, so the gradient signal cleanly reflects how rollout i compared to the rest of its group. The REINFORCE part means the policy gradient is the advantage-weighted log-probability of each rollout, the standard policy-gradient form. There is no separate critic network learned alongside the policy. The group of rollouts plays the critic's role through their leave-one-out averages. The other options each describe a real but different baseline. Option B is PPO-style actor-critic. Option C is a moving-average baseline that vanilla REINFORCE variants sometimes use. It has the bias problem of mixing stale and current reward magnitudes. Option D uses the group max, so the best rollout gets zero advantage and every other rollout gets a negative one. Nothing is ever pushed up. RLOO matches PPO on RLHF benchmarks with simpler infrastructure (no critic to tune, no value-loss head to maintain). That is why it has become a common drop-in replacement.

TL;DR

RLOO generates k rollouts per prompt and uses the mean of the other k-1 as each rollout's baseline. It needs no critic, keeps each rollout out of its own baseline, and gives unbiased advantages.

Memory aid

Easy to grasp

Imagine four students each solve the same math problem, and a teacher scores all four. To decide how good Alice's solution is compared to the rest, you average the scores of the other three students and compare Alice to that average. You do not include Alice's own score in the average, because that would bias the comparison toward her. Then you do the same for Bob, comparing him to the average of the other three. That tiny statistical trick is the whole heart of RLOO. The model generates several attempts at the same prompt and scores them. It learns more from the attempts that beat the typical attempt and less from the ones that lagged behind.

Key concepts

RLOO baseline RLHF REINFORCE policy gradient Advantage estimation

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

RLOO solves a small problem with a clean trick. Policy gradient methods need a baseline to reduce gradient variance. PPO trains a separate critic network to produce that baseline, which works well but adds infrastructure: a value head, a separate loss, clipping, GAE. RLOO replaces all of that with a statistical operation over a small group of rollouts. It takes k rollouts per prompt and uses the leave-one-out mean as each rollout's baseline.

The name is literal. For rollout i, the baseline is the average reward of the other k-1 rollouts in the same group. The rollout's own reward is left out, which is what keeps the baseline unbiased and the gradient clean.

The payoff is that RLOO matches PPO on standard RLHF benchmarks with substantially simpler infrastructure. There is no critic network to train, no value loss to balance against the policy loss, no clipping epsilon to tune. The algorithm fits on a page and holds no critic weights or critic optimizer state in memory.

The leave-one-out formula

Start from the policy-gradient form. For each prompt, the policy generates k rollouts. The reward model scores each rollout, giving rewards r_1 through r_k. The policy gradient is the advantage-weighted log-probability of each rollout, summed across the group.

The advantage for rollout i is its reward minus a baseline. The baseline should be a reasonable estimate of the expected reward at this prompt, independent of rollout i's own action. RLOO computes the baseline as the mean of the other k-1 rewards:

A_i = r_i - \frac{1}{k-1} \sum_{j \neq i} r_j

The sum explicitly excludes index i. With k=4 and rewards [0.8, 0.5, 0.6, 0.7], the baseline for rollout 1 is (0.5 + 0.6 + 0.7) / 3 = 0.6, and the advantage is 0.8 - 0.6 = 0.2. For rollout 2 the baseline is (0.8 + 0.6 + 0.7) / 3 = 0.7, and the advantage is 0.5 - 0.7 = -0.2.

Each rollout has its own baseline, and that baseline depends only on the other rollouts in its group. The advantages of the k rollouts sum to zero by construction. So the policy gradient sees both positive (above the group's leave-one-out average) and negative (below it) signals, balanced symmetrically. This is the variance-reduction property baselines are supposed to deliver.

Why the exclusion matters

How RLOO replaces the critic in PPO

Distinguishing RLOO from best-of-k and from running-average baselines

Group size k as a trade-off knob

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Method	Baseline source	Critic network?	Infrastructure complexity
PPO (actor-critic)	Learned value function	Yes (trained jointly)	High; value head, clipping, GAE
RLOO	Mean of other k-1 rollouts in same group	No	Low; just rollouts + REINFORCE
Vanilla REINFORCE	Running average or none	No	Lowest but highest variance
RAFT / expert iteration	Best-of-k filtering, supervised on top	No	Low; not a policy gradient

Real products, models, and research that use this idea.

Hugging Face TRL implements RLOOTrainer alongside DPOTrainer and GRPOTrainer. Current TRL sets the group size with num_generations; the older rloo_k parameter was deprecated in v0.22.0 and later removed.
ReMax, another critic-free REINFORCE method, uses the reward of the greedy response as its baseline and reports about 46% GPU memory saving versus PPO for a 7B model.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does RLOO differ from GRPO, and when would you pick each?

GRPO (Group Relative Policy Optimization) also generates k rollouts per prompt but normalizes advantages by the group's standard deviation, scaling per-prompt variability. RLOO subtracts the leave-one-out mean without standardization. GRPO's scaling keeps the signal comparable across prompts; RLOO is simpler when reward variance is well-behaved.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Including rollout i's own reward in the baseline (the group mean of all k), which biases the advantage toward zero and reduces signal. Leave-One-Out excludes the rollout's own score by definition.

60 second bullets to scan on the way to the call.

The exact formula for the RLOO baseline as the mean of the other k-1 rewards
Why excluding the rollout's own reward keeps the baseline unbiased

Primary sources. Browse if you want the original framing.