A team is running multi-week GRPO post-training on a 70B reasoning model with sparse verifiable rewards. Explain why vanilla GRPO can destabilize at this scale and name concrete DAPO style fixes that keep the run healthy.
Vanilla GRPO destabilizes on long sparse-reward runs via advantage collapse, KL drift, and high-variance updates — DAPO-style fixes use dynamic sampling, clip tweaks, and reward shaping.
GRPO compares several answers to the same question and learns from which did better than average. On a month-long training run with sparse right/wrong scores, every answer in a group might tie wrong — then nobody learns. The model may also ramble longer to game rewards or drift far from the original model. DAPO-style fixes resample until groups have spread, clip wild updates, and penalize length hacks so advantages stay meaningful step after step.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
GRPO's critic-free elegance breaks in the messy middle of 70B multi-week RLVR — exactly when reasoning models are trained. Interviewers at senior level expect you to name advantage collapse, KL drift, and DAPO-style mitigations, not just the group mean formula.
Vanilla math is correct; operational stability is not automatic.
The sections below build mechanism first, then production tradeoffs, then how teams measure success in 2026 deployments. Read each heading as a promise — by the end you should explain this topic to a colleague designing a reasoning API or post-training run.
Vanilla GRPO failure: advantage collapse
GRPO advantages need spread in . On hard math verifiers, all K completions fail frequently — identical rewards, , advantages vanish. Same for all-pass easy batches.
Over thousands of steps, a high zero-advantage rate stalls learning while burning rollout FLOPs. This is the dominant practical instability on sparse RLVR, not subtle numerics.
Log zero-advantage step rate daily on 70B runs. Healthy runs stay below 20–30% on hard verifiers; above 40% means dynamic sampling or curriculum adjustment is overdue.
Curriculum scheduling reduces early tie rate — start RLVR on easier verifier subsets, anneal toward competition hardness as pass rate stabilizes.
Production checkpoint. Before shipping, walk through a concrete scenario with real numbers: who owns the metric dashboard, what fails first under load, and what you would change after one week of live traffic. Interviewers reward answers that connect mechanism to operability — not only definitions. If you can name one 2026 vendor example and one failure mode for this topic, you are already ahead of candidates who stop at textbook recitation.
Production checkpoint. Before shipping, walk through a concrete scenario with real numbers: who owns the metric dashboard, what fails first under load, and what you would change after one week of live traffic. Interviewers reward answers that connect mechanism to operability — not only definitions. If you can name one 2026 vendor example and one failure mode for this topic, you are already ahead of candidates who stop at textbook recitation.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DAPO paper and DeepSeek follow-on work document dynamic sampling for GRPO stability
- DeepSeek-R1 training reports multi-week GRPO with KL control and hacking mitigations
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect advantage collapse in training logs?
Track fraction of groups with σ_G≈0, mean |A|, step count with no gradient signal — alert thresholds.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Describing GRPO as inherently stable at 70B scale without naming tie collapse, KL drift, or DAPO-style diversity fixes.
60 second bullets to scan on the way to the call.
Name advantage collapse on tied sparse group rewards
Mention KL drift or policy collapse over long runs
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.