Predict what changes when you raise GRPO group size from 4 to 16

MCQ·Medium·4.0 · 0·~1 min·Asked atCohere Jpmorgan Replicate

Attempt it

**The right answer is A.** Larger groups give a better-estimated per-prompt baseline, which lowers the variance of the policy gradient, at a roughly 4x rollout cost going from 4 to 16. GRPO computes each rollout's advantage by subtracting the group mean reward and dividing by the group standard deviation: $$A_i = \frac{r_i - \bar{r}}{\sigma + \epsilon}$$ When the group has only 4 samples, the mean and std are noisy estimates. A single lucky or unlucky rollout shifts the baseline noticeably. With 16 samples, the mean and std are much more stable, so each rollout's advantage is a more reliable measure of how much better or worse it is than typical for that prompt. **Why the other options are wrong.** B is wrong because the KL penalty is a separate term independent of group size. C is wrong because GRPO has no value network, that is its defining feature versus PPO. D is wrong because group size by itself does not guarantee diversity; if the policy has collapsed to a single mode, sampling 16 times still produces 16 near-identical rollouts. The price for the variance reduction is linear in rollouts: 4x the generations and 4x the reward-model evaluations per prompt.

**Answer A.** Raising GRPO group size from 4 to 16 reduces the variance of the group-baseline estimate (mean and std), which gives lower-variance, higher-quality advantage estimates. The cost is linear in rollouts: roughly 4x generations and reward evaluations per prompt. GRPO's defining design choice is to use the group itself as the baseline, replacing PPO's learned value network: $$A_i = \frac{r_i - \bar{r}}{\sigma + \epsilon}, \quad \bar{r} = \frac{1}{G} \sum_i r_i$$ The quality of `r-bar` and sigma as estimators improves with the square root of G. At G=4 they are noisy point estimates; at G=16 they are reasonably stable. Lower-variance advantages translate directly into more reliable gradient direction per step, which usually shows up as faster convergence per gradient step and a lower fraction of wasted updates. **Why the distractors fail.** - **B (KL shrinks).** The KL term is computed per-token against the reference policy and has nothing to do with group size. Raising G does not loosen the leash. - **C (scale up value network).** GRPO has no value network at all; that is the whole point of the algorithm relative to PPO. - **D (eliminates mode collapse).** Group size does not prevent collapse. A collapsed policy produces near-identical rollouts at any G; you need temperature, entropy regularization, or curriculum to address collapse. **Practical band.** Production GRPO recipes (DeepSeek-R1, open-weight RLVR) typically run G between 8 and 16. Past G=16 the variance gains saturate while the rollout cost keeps growing linearly.

TL;DR

Option A. Larger groups produce a lower-variance group baseline and more reliable advantage estimates, paid linearly in rollout compute per prompt.

Memory aid

Easy to grasp

Imagine you want to know if a kid did well on a quiz by comparing them to their classmates. With four classmates, one weird score swings the average a lot, so the comparison is noisy. With sixteen classmates, the average is much steadier and the kid's score is easier to place fairly. That is GRPO's group baseline: more rollouts make the per-prompt average steadier, so each rollout's relative ranking is more trustworthy. The cost is that you have to grade sixteen quizzes instead of four every round.

Key concepts

RLHF

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

GRPO is built around one design choice: replace PPO's value network with a baseline computed from the rollout group itself. That choice halves the trained-parameter footprint and removes a long list of engineering headaches. It also makes group size a first-class hyperparameter, because the baseline's quality is determined by how many rollouts you average over.

The question asks what changes when you raise group size from 4 to 16. The right answer is the one that names the variance-reduction effect and the linear compute cost. The other options test for common misconceptions: that GRPO has a value network, that KL scales with group size, or that more rollouts somehow fix mode collapse. None of these are true.

This deep dive walks through the variance argument quantitatively, why the distractors fail, and how production teams choose G in real recipes.

Why the group is GRPO's baseline, and what variance scaling means

GRPO computes each rollout's advantage by subtracting the group mean reward and dividing by the group standard deviation:

A_i = \frac{r_i - \bar{r}}{\sigma + \epsilon}

The statistic r-bar is a sample mean of G independent rewards. Its standard error scales as the population standard deviation divided by sqrt(G). Going from G=4 to G=16 doubles sqrt(G), which halves the standard error of the baseline estimate. The standard deviation estimate similarly improves, with the joint effect of giving each advantage A_i a substantially smaller estimation-noise floor.

Why this matters for the gradient: the policy-gradient direction is a sum of A_i times log-probability gradients over rollouts. Noise in A_i becomes noise in the gradient. Halving the baseline noise reduces the gradient variance noticeably, especially for prompts where the underlying reward distribution is wide. Lower-variance gradients mean more reliable updates per step, which usually shows up as faster convergence per token of training compute.

The cost is linear: 4x rollouts and 4x reward-model evaluations per prompt going from G=4 to G=16. Whether the variance reduction is worth the rollout cost depends on the regime; in modern RLVR with cheap rule-based rewards, the answer is usually yes.

The variance argument is mechanical. The GRPO baseline is $\mu_g = \frac{1}{G} \sum_i r_i$ , an empirical mean over G samples. Variance of that mean is $\sigma^2/G$ where $\sigma^2$ is the reward variance per rollout. Quadrupling G (4 to 16) cuts baseline variance to one-quarter, which translates into roughly halved standard error on the advantage estimate. Halved gradient noise means you can take twice the effective step size at the same training stability, or hold step size constant and converge faster.

Why option B is wrong — KL is independent of group size

Why options C and D are wrong — value network and mode collapse

How production recipes choose G in practice

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

DeepSeek-R1's GRPO runs use group sizes in the 8-16 range to balance variance reduction against rollout cost on math tasks
Kimi K1.5 reports its RL group size as a primary tuning lever for stability of long-context reasoning training

What an interviewer would ask next. Try answering before peeking at the approach.

QWould you ever use G=2, and what does the algorithm degenerate to at that limit?

Discuss how G=2 collapses GRPO toward pairwise-contrast advantages, similar in spirit to DPO-style pair training, with much higher gradient variance per step.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Believing GRPO uses a value network like PPO, or that larger groups change the KL term or guarantee response diversity.

60 second bullets to scan on the way to the call.

How GRPO computes advantages from group mean and std
Why GRPO has no value network

Primary sources. Browse if you want the original framing.