CoT distillation and STaR: what made rejection sampled reasoning FT effective?
Explain how chain-of-thought (CoT) distillation works as a fine-tuning technique. What did STaR (Self-Taught Reasoner, Zelikman et al. 2022) add, and why is rejection sampling on OUTCOME the load bearing trick?
CoT distillation fine-tunes a student on a teacher's reasoning traces. STaR drops the teacher and keeps only the model's own correct-answer traces, verified cheaply by outcome.
Imagine teaching a kid math by showing worked solutions, not just answers. The kid copies how you reason, not what you memorise. That is CoT distillation: a strong model writes out its thinking, and a smaller model trains on that thinking. STaR is the clever twist where the kid teaches itself. It tries lots of problems, and you keep only the worked solutions that reached the right answer. You never grade the reasoning itself, just the final answer, which is easy to check. Over many rounds the kid keeps the lucky-and-correct work, retrains on it, and slowly gets genuinely better at reasoning, all without a smarter teacher in the loop.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Reasoning fine-tuning sits on one deceptively simple idea: you can teach a model to reason by training it on examples of reasoning, the same way supervised fine-tuning teaches any other behavior. The reasoning trace is just another sequence to predict. What makes the topic interview-worthy is not the supervised step, which is ordinary, but where the high-quality reasoning traces come from and how you decide which ones to keep.
Chain-of-thought distillation is the baseline answer. A capable teacher writes out step-by-step solutions, and a student fine-tunes on the question, the trace, and the final answer together. The student inherits the teacher's habit of thinking before committing. The ceiling, though, is the teacher: you need a model already good at the reasoning you want, and the student rarely exceeds it.
STaR, the Self-Taught Reasoner, removed that ceiling with a bootstrap. The model generates its own traces, you keep only the ones that reached the correct answer, you fine-tune on the survivors, and you repeat. No stronger teacher is required. The genius is in the filter, and that filter is the thing every interviewer is probing for. This deep dive walks the mechanism, the cheap-verifier argument, the false-positive risk, the controversial rationalisation step, and the line that runs from STaR to today's reasoning models.
CoT distillation: reasoning as a sequence to predict
Chain-of-thought distillation is supervised fine-tuning where each example carries the reasoning, not just the answer. A teacher model solves problems with known answers and emits a full trace. Every training row becomes a triple of question, reasoning steps, and final answer. The teacher can be a genuinely larger model, a more heavily aligned sibling, or simply the same model run with a stronger prompt and more samples.
The student trains to predict the trace and the answer together, typically as one continuation after the prompt. Loss falls on both the reasoning tokens and the answer tokens, so the model learns the structure of stepping toward a conclusion, not only the conclusion itself. Formatting matters here in practice: you fix a delimiter between the trace and the final answer so the model learns to think, then commit cleanly, and so your harness can parse the answer back out for evaluation.
The value over answer-only fine-tuning is generalisation. A model trained to emit intermediate steps decomposes new problems instead of pattern-matching a single final answer. The intermediate steps act as scratch space, letting the model spread computation across tokens rather than forcing a one-shot leap to the answer. That is why even a small student often gains disproportionately from distilled reasoning: it learns to use its own forward pass more like a worked calculation than a guess.
The hard limit is the teacher. You must already own a model that reasons well in your domain, and the student usually lands at or below the teacher's quality. You also pay for teacher inference on every problem, and any systematic blind spot in the teacher is copied faithfully into the student. That ceiling, and that dependency on a pre-existing strong teacher, is exactly what STaR was built to break.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | Teacher CoT distillation | STaR self-bootstrap |
|---|---|---|
| Trace source | Stronger external teacher model | The model's own sampled traces |
| Filter signal | Trust the teacher, optionally outcome-check | Outcome correctness only, iterated |
| Verifier cost | None beyond teacher inference | Cheap ground-truth answer check |
| Main failure mode | Student capped by teacher quality | False positives: right answer, wrong reasoning |
| Modern descendant | Synthetic SFT data generation | Rejection-fine-tuning, R1-style outcome RL |
Real products, models, and research that use this idea.
- DeepSeek-R1 popularised outcome-reward reinforcement learning on verifiable math and code, the direct descendant of STaR-style outcome filtering at scale.
- Rejection fine-tuning pipelines sample many traces from a model like Llama 4, keep only the answer-correct ones, and fine-tune on that filtered set.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do false positives, where a flawed trace reaches the right answer, affect the bootstrapped model over many STaR iterations?
Reason about the statistics: wrong reasoning produces correct answers at a low, roughly random rate, so correct reasoning dominates the kept set and the signal-to-noise ratio improves each round.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking the model is filtered on reasoning quality. It is filtered only on the final answer; the trace is never graded, which is exactly what makes the verifier cheap.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.