Spot the errors in this plan to distill reasoning into a 7B student

Spot the error·Hard·4.0 · 0·~2 min·Asked atComet Ml Zed

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

The plan contains three confident assertions. Each one is the opposite of what the DeepSeek-R1 team reported when they ran the experiment. First, supervised imitation on reasoning traces is not memorization. The traces are long, varied, and include backtracking and self-correction. SFT on them teaches the student the *behavior*, not specific answers. The 800,000-sample trace dataset used in the R1 distillation includes problems the student will never see at eval time. Second, small models do not discover long-chain reasoning under RL with verifiable rewards. The signal RL needs (occasional correct long chains by accident) almost never appears at 7B scale on hard problems. The R1 team tested this directly with Qwen-32B-Base and reported that distillation beat direct RL. Third, imitation learning is bounded above by the teacher distribution. The student fits the teacher's traces. It does not invent better traces. Surpassing the teacher needs one of two things: a new RL run with verifier signal beyond what the traces encode, or new high-quality data the teacher did not produce. Expecting the student to surpass the teacher by default is wrong.

**Each clause reverses a published finding from the R1 distillation work.** Clause one calls SFT on teacher traces 'just memorization' and proposes skipping it. The R1 release made the opposite case empirically: the distilled SFT-only checkpoints (Qwen-7B, Llama-8B, Qwen-32B) reached AIME and MATH numbers that beat much larger non-reasoning models. The behavior being installed is the long-deliberation policy: when to verify, when to backtrack, when to re-plan. That is a policy transfer, not a lookup-table memorization. Clause two claims direct RL on the 7B is just as good. The R1 paper ran that ablation on a 32B base. Large-scale RL on Qwen-32B-Base underperformed the trace-distilled Qwen-32B. The reason is structural: the success-trace rate at small scale is too low for the verifier reward to drive learning. RL needs a base strong enough that long sampled chains occasionally hit the right answer. Clause three predicts the student surpasses its teacher on AIME. Imitation learning is bounded above by the teacher's policy in distribution. The student can approach the teacher and occasionally beat it on a noisy benchmark slice. Systematic surpassing needs added signal: new data, new RL with rewards beyond the trace distribution, or test-time scaling. Without those, expecting the student to exceed the teacher is wishful. The corrected plan: SFT on curated teacher traces first, then optional GRPO refinement on top, with explicit acknowledgement that the teacher is the imitation ceiling.

TL;DR

All three claims invert published findings: trace SFT is the proven recipe, direct RL on small models underperforms distillation, and the student is bounded near the teacher unless extended with new data or RL.

Memory aid

Easy to grasp

Think of cooking school. Claim one says watching a master chef and copying is just memorizing recipes. In reality, that is how you learn the craft. Claim two says a beginner with a stove and a cookbook will figure out new dishes on their own as well as the master did. In reality, the beginner needs the demonstrations first. Claim three says the student will end up a better chef than their teacher just by copying. In reality, copying gets you close to the teacher but not past them. Each claim sounds confident, but each one inverts what actually happened when researchers tried it on real reasoning models.

Key concepts

Reasoning Models

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Reasoning distillation has accumulated confident folk wisdom that is the opposite of what the published experiments showed. The plan is a compressed version of that folk wisdom: three confident clauses, each one contradicted by the DeepSeek-R1 ablation table.

Clause one: SFT on traces is not memorization

The plan dismisses trace SFT as 'supervised imitation only memorizes.' This frames imitation learning as a lookup table, which mischaracterizes both the data and the training objective.

Reasoning traces are long, conditional on the input problem, and structurally varied. A single problem can yield ten different valid traces with different branching, backtracking, and verification patterns. Memorization would require the model to store the mapping from input to output verbatim. The traces are often 2,000 to 20,000 tokens long, and the datasets hold hundreds of thousands of traces. At these sizes, the model does not have the parameter count for verbatim storage of more than a small fraction of the data. What it stores is the conditional pattern.

The held-out evaluation backs this up. The R1 distilled 7B reaches strong AIME 2024 numbers. But AIME 2024 problems were not in the teacher's training set or the distillation trace set in any verbatim form. If the model were memorizing, those held-out scores would be near zero. They are not. They beat much larger non-reasoning models.

The correct mental model is policy transfer. The teacher discovered a reasoning policy through expensive RL. The traces are samples from that policy. SFT compresses the policy into the student's weights by next-token prediction. The student then applies the policy to new inputs, including held-out benchmarks.

Diagnostic check: if the plan's claim were true, distilled-student AIME numbers should drop sharply on held-out years (AIME 2025 or 2026 problems released after the trace collection). They do not. They degrade only modestly, which is the signature of policy transfer rather than memorization.

Clause two: small models do not discover reasoning under direct RL

Clause three: imitation is bounded by the teacher

The corrected plan and how to defend it

Common false claims about reasoning distillation

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

DeepSeek-R1-Distill-Qwen-7B and -Llama-8B (January 2025) ship as SFT-only distilled reasoning models. Their numbers refute the plan empirically.
The s1 paper (Muennighoff et al., 2025) showed that test-time scaling (budget forcing) on top of distillation lifts accuracy beyond pure SFT. It brought its 32B student close to its teacher.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you detect the moment in training where GRPO advantage signal collapses to noise on a small base?

Track per-group reward variance and the fraction of groups containing at least one correct rollout. Once that fraction drops below a threshold (often 10 to 20 percent), the gradient is mostly entropy-driven and you should fall back to SFT or curriculum simplification.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Reading the plan as internally consistent because each clause sounds intuitive in isolation. The published ablation contradicts each one.

60 second bullets to scan on the way to the call.

Why SFT on reasoning traces is policy transfer, not memorization
What the R1 paper's direct-RL ablation on a 32B base showed

Primary sources. Browse if you want the original framing.