Explain how chain-of-thought (CoT) distillation works as a fine-tuning technique. What did STaR (Self-Taught Reasoner, Zelikman et al. 2022) add, and why is rejection sampling on OUTCOME the load-bearing trick?
Free · 2 AI evals / day
**Chain-of-thought distillation** teaches a model to reason by training it on worked solutions, not just answers. A strong teacher solves problems out loud, step by step, and a smaller student model trains on those step-by-step traces plus the final answer.
Each training example is a triple: the question, the reasoning steps, and the answer. The student learns to imitate the pattern: think first, then commit. It inherits the teacher's reasoning habits without needing to be as large.
STaR, the Self-Taught Reasoner, removes the need for a smarter teacher. The model generates its own reasoning traces for problems where the answer is known. You keep only the traces that reached the correct answer and throw the rest away. Then you fine-tune on the kept set and repeat the loop.
The key trick is what you check. You never grade the reasoning itself; you only check the final answer. That is cheap for math, code, or facts with known answers. Wrong reasoning rarely lands the right answer often, so over many rounds the kept traces are mostly good reasoning.
**CoT distillation** is supervised fine-tuning on reasoning triples. A teacher produces step-by-step traces for problems with known answers. Each example becomes a question, a trace, and a final answer. The student trains to predict the trace and answer together, inheriting the teacher's reasoning style.
STaR removes the stronger teacher. The model samples its own traces for problems with known answers. You keep only traces that reach the correct answer, then fine-tune and repeat. This is a self-improvement loop bootstrapping reasoning from the model's own lucky-correct outputs.
The trick that makes the loop work is outcome-based rejection sampling. Judging trace quality needs a separate, biased, hard-to-build evaluator. Checking the final answer needs only ground truth, which is cheap and unambiguous for math, code, and many factual tasks. The catch is false positives: a right answer from flawed reasoning. Across many iterations this noise averages out, because wrong reasoning rarely lands the right answer consistently. STaR also added a backward rationalisation step for failed problems, generating a trace given the answer, which is more controversial. Modern descendants include rejection fine-tuning and DeepSeek-R1-style outcome reinforcement learning with verifier rewards.
Quick reference
TL;DR
CoT distillation fine-tunes a student on a teacher's reasoning traces. STaR drops the teacher and keeps only the model's own correct-answer traces, verified cheaply by outcome.
Imagine teaching a kid math by showing worked solutions, not just answers. The kid copies how you reason, not what you memorise. That is CoT distillation: a strong model writes out its thinking, and a smaller model trains on that thinking. STaR is the clever twist where the kid teaches itself. It tries lots of problems, and you keep only the worked solutions that reached the right answer. You never grade the reasoning itself, just the final answer, which is easy to check. Over many rounds the kid keeps the lucky-and-correct work, retrains on it, and slowly gets genuinely better at reasoning, all without a smarter teacher in the loop.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tasks with no cheap automatic check on the final answer, such as open-ended essays Outcome rejection sampling needs an unambiguous ground-truth answer; without one the filter collapses.
Domains where lucky-correct answers from flawed reasoning are common, like easy multiple choice Guessing inflates outcome correctness, so the kept set fills with bad reasoning and the loop drifts.
When you already have a strong, trusted teacher and abundant labelled traces Plain distillation is simpler and avoids the iteration overhead and false-positive contamination of bootstrapping.
Very small base models that almost never reach correct answers on the target problems The kept set is too sparse to bootstrap; cold-start the model first or seed it with teacher traces.
Quick scan · 2–4 min
Everything important, quickly.
5 min: CoT distillation as triple supervised tuning, then the STaR self-bootstrap loop, why outcome filtering beats trace grading, the false-positive statistics, the rationalisation caveat, and the lineage to rejection-fine-tuning and R1-style outcome RL.
Side-by-side
Aspect
Teacher CoT distillation
STaR self-bootstrap
Trace source
Stronger external teacher model
The model's own sampled traces
Filter signal
Trust the teacher, optionally outcome-check
Outcome correctness only, iterated
Verifier cost
None beyond teacher inference
Cheap ground-truth answer check
Main failure mode
Student capped by teacher quality
False positives: right answer, wrong reasoning
Modern descendant
Synthetic SFT data generation
Rejection-fine-tuning, R1-style outcome RL
Real-world examples
Real products, models, and research that use this idea.
1DeepSeek-R1 popularised outcome-reward reinforcement learning on verifiable math and code, the direct descendant of STaR-style outcome filtering at scale.
2Rejection fine-tuning pipelines sample many traces from a model like Llama 4, keep only the answer-correct ones, and fine-tune on that filtered set.
3Frontier reasoning models such as GPT-5.5 and Gemini 3.1 Pro are widely understood to use verifier-filtered chain-of-thought traces during post-training.
4Open recipes on Hugging Face TRL and Unsloth ship rejection-sampling plus SFT loops that reproduce the STaR bootstrap on GSM8K and MATH-style benchmarks.
5Claude Opus 4.7 and similar systems pair outcome verifiers with long-form reasoning traces to distill stronger step-by-step problem solving into deployable models.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do false positives, where a flawed trace reaches the right answer, affect the bootstrapped model over many STaR iterations?
Sign in or go Pro to see the approach for this follow-up.
QWhy does the rationalisation step risk teaching the model to confabulate, and when would you still use it?
Sign in or go Pro to see the approach for this follow-up.
QHow does DeepSeek-R1-style outcome-reward RL generalise the STaR loop, and what does the verifier become?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Thinking the model is filtered on reasoning quality. It is filtered only on the final answer; the trace is never graded, which is exactly what makes the verifier cheap.