Every answer now opens with the same three phrases: name the failure

MCQ·Medium·4.0 · 0·~1 min·Asked atCopy Ai Ironclad Snap

Attempt it

The pattern of identical openings across answers is the textbook signature of **mode collapse**. Here is what happens. RL optimization gives the policy a reward signal and asks it to climb. If a few opening phrases (say, 'Great question. Let me break this down. First,') reliably score well, the policy concentrates probability mass on those phrases. Each PPO step nudges the next-token distribution sharper around them. Policy entropy falls. The variety the reference model had is still computable, but it is now drowned out by the high-reward attractor. The other options describe failures with different fingerprints. A tokenizer change does not collapse diversity globally. A KL coefficient that is too high keeps the policy too close to the reference, preserving diversity rather than killing it. Missing varied openings in preference data would mean the model never had diverse openers to begin with, but the question implies the model started varied and lost variety. Fixes are straightforward to name: strengthen the KL anchor, add an entropy bonus, stop training earlier, or diversify the reward signal so no single template dominates.

**Answer: A.** Identical openers across responses is the canonical fingerprint of mode collapse from reward overoptimization. **Mechanism.** The reward model rates a few opening templates highly. PPO concentrates probability mass on whatever scores best. As training proceeds, the policy's next-token distribution becomes sharper around the winning templates and policy entropy falls. The reference model's diversity is no longer expressed in the rollouts because the gradient keeps pulling away from it. The visible symptom is stereotyped openers, often paired with stereotyped structure (always three bullets, always the same hedge). **Why the distractors fail.** - **Tokenizer merge.** Tokenizers are fixed during training. Even if a phrase became one token, that would not cause it to dominate across all answers. - **KL too high.** This is the opposite direction. A high KL coefficient pins the policy near the reference distribution and preserves its diversity. Mode collapse happens when KL is too weak to anchor the policy against the reward attractor. - **No varied openings in preference data.** This would mean the model never saw variety, but the question implies the model had it (pre-RL) and lost it. The dynamic is loss of diversity, not absence of it. **Fix.** Increase the KL coefficient, add an entropy bonus to the loss, run an early-stopping protocol gated on a diversity metric (distinct-n on opener n-grams), and diversify the reward signal so no single template can dominate.

TL;DR

Identical openers across answers is the fingerprint of mode collapse: reward overoptimization collapses policy entropy onto a few high-scoring templates.

Memory aid

Easy to grasp

Imagine a stand-up comic who notices one joke always gets a laugh. They start opening every set with that joke. Then they add the next safe bet. Soon every show begins the same way, and the audience can recite the first three minutes. Nothing is technically wrong with the routine, but the comedian has lost their range. The same thing happens to a model under reinforcement learning. When the reward keeps paying out for a particular opener, the model leans on it harder and harder, until that opener becomes the default. The variety in the reference model is still in there somewhere; it just stopped winning the gradient.

Key concepts

RLHF

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Mode collapse is the most visible failure mode of RLHF after length inflation, and it is the one most likely to be misdiagnosed because three of its plausible causes (tokenizer, KL, data) point in directions that are either irrelevant or backwards. A model whose answers all open the same way is not broken in some new way; it is doing exactly what an overoptimized policy is supposed to do, which is concentrate mass on the highest-scoring patterns.

This deep dive walks through the entropy-collapse mechanism, why each plausible-sounding distractor inverts a lever or confuses an axis, the monitoring discipline that catches collapse early, and the toolkit of fixes that has consolidated in 2026 production practice.

The goal is to leave you able to read the symptom (identical openers, stereotyped structure, monotone refusals) and trace it back to its source (entropy collapse from reward overoptimization with insufficient KL anchoring) without taking a detour through the distractors.

Why identical openers means entropy collapsed

PPO updates the policy to climb a reward landscape under a KL constraint. The landscape is shaped by the reward model. If a few opening templates reliably score well, every PPO step shifts probability mass toward those templates. The policy's next-token distribution at the start of each response becomes sharper, and the entropy of that distribution falls.

This is not an artifact, it is the optimization working. The reward model is a learned approximation of preferences. Approximations have peaks. Optimization climbs peaks. The peaks are narrow, so the policy ends up narrow.

The variety in the reference model is still computable in principle, but the gradient is no longer pulling toward it. Without an anchor, the policy can drift arbitrarily far from the reference distribution. The visible result is stereotypy, in openers, in structure, in hedging language, in length, in any dimension where the reward has a peak narrower than the reference's spread.

Make entropy collapse measurable. Sample 100 responses to 100 held-out prompts at the same temperature. Compute the unique-prefix rate (fraction of unique first five token sequences) and the bigram diversity (number of distinct token bigrams normalized by total). A healthy RLHF policy maintains 85 to 95 percent prefix uniqueness and high bigram diversity. A mode-collapsed policy drops to 30 to 60 percent prefix uniqueness, with a handful of openers dominating. The repeated three-phrase openings are a literal observation of this collapse on the prefix axis.

Why each distractor is a diagnostic error worth understanding

Monitoring and early detection

The fix toolkit, in order of escalation

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Early InstructGPT runs reported telltale stereotypy ('As an AI language model, I cannot...') that stemmed from RM overoptimization on a narrow refusal template
Claude Opus 4.7's training notes describe entropy and diversity monitoring during constitutional RLAIF rounds, with explicit early-stopping gates

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you set up a diversity monitor that catches collapse before users notice it?

Track distinct-n on response openers across a held-out prompt set every K steps, plot policy entropy alongside reward, and trip alerts on relative drops rather than absolute thresholds.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Inverting the KL coefficient's direction. A high KL keeps the policy diverse; a low KL lets it collapse onto high-reward templates.

60 second bullets to scan on the way to the call.

What policy entropy measures and why it matters during RLHF
How KL coefficient choice affects diversity and reward gain

Primary sources. Browse if you want the original framing.