Explain the difference between classical logit distillation and FT-based distillation. Which one works when your teacher is a closed-API model (GPT-4o, Claude), and why?
Free · 2 AI evals / day
**Knowledge distillation** trains a small student model to copy a bigger, smarter teacher. There are two common ways to do it, and they differ in what they copy.
Logit distillation copies the teacher's full opinion. At each token the teacher produces a probability over the whole vocabulary, not just one winning word. The student is trained to match that whole distribution. This carries 'soft' information: how confident the teacher was, and which runner-up words it almost picked. The loss is KL divergence between the two distributions.
FT-based distillation copies only the teacher's finished answers. The teacher generates text, and you do ordinary supervised fine-tuning on those (prompt, answer) pairs. The student never sees the soft distribution, only the words that actually came out. The loss is the usual next-token cross-entropy.
Why does this matter with closed APIs? To run logit distillation you need the teacher's per-token probabilities. GPT-4o, Claude, and Gemini do not hand those over. So logit distillation is off the table for them. FT-based distillation only needs the teacher to write text, which every API does, so that is the method people actually use.
Both methods shrink a big teacher into a small student. They differ in the signal they extract.
**Logit distillation** trains the student to match the teacher's full next-token distribution at every position. The loss is KL divergence between the two softmaxes over the vocabulary. This transfers soft knowledge: the teacher's confidence and the alternatives it nearly chose. Computing that loss requires the teacher's per-token logits or probabilities.
FT-based distillation needs only generated text. The teacher writes responses, and you run standard supervised fine-tuning on the resulting prompt-response pairs. The student imitates the argmax behavior, not the soft distribution. The loss is plain next-token cross-entropy.
Closed-API teachers settle the choice. Logit distillation needs the full vocab softmax. Closed APIs expose at most top-k logprobs, often capped or absent. So you cannot compute the KL term for a GPT-4o, Claude, or Gemini teacher.
FT-based distillation works with any teacher you can query, since generation is all every API offers. You sample thousands to tens of thousands of pairs, then SFT the student. The signal is lossier; you discard the rejected-alternative information. It is the production standard for closed-API teachers. Teams usually add rejection sampling: generate several candidates, keep those passing a quality check, then SFT on the survivors.
Quick reference
TL;DR
Logit distillation matches the teacher's full softmax via KL and needs its logits. FT-based distillation just SFTs the student on teacher-generated text, so it works with any closed-API teacher.
Imagine learning chess from a grandmaster. The richest way is the master narrating not just the move they played, but how much they liked every other move too, a full ranked opinion on each turn. You copy that whole ranking. But some masters will only let you watch them play; they never share the rankings inside their head. So you fall back to the simpler way: watch thousands of their finished games and learn to imitate the moves you saw. You lose the rich 'how sure were they' signal, but you can learn from any master you can watch. To get cleaner lessons, you keep only their best games and study those.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Teacher is a closed-API model and you insist on logit distillation The full vocabulary softmax is never exposed, so the KL loss cannot be computed at all.
Teacher and student come from different tokenizer families Logit targets do not align position to position, so soft-target transfer is ill-defined without remapping.
The task needs fresh or private facts the teacher never saw Distillation copies behavior, not knowledge; missing facts belong in retrieval, not in the student's weights.
The teacher's license forbids training a competing model on its outputs FT-based distillation on those generations becomes a legal liability, not just a technical choice.
Quick scan · 2–4 min
Everything important, quickly.
4 min: define both methods + the KL vs cross-entropy loss + what closed APIs expose + why FT-based is the only option there + rejection sampling and CoT traces + the licensing caveat.
Side-by-side
Aspect
Logit distillation
FT-based distillation
Signal copied
Full softmax over vocab (soft targets)
Sampled text only (hard targets)
Loss
KL divergence between distributions
Next-token cross-entropy (standard SFT)
Teacher access needed
Per-token logits plus aligned tokenizer
Text generation only
Works with closed APIs
No (full logits not exposed)
Yes (any callable teacher)
Information transferred
Confidence and rejected alternatives
Argmax behavior only (lossier)
Common augmentation
Temperature scaling on teacher
Rejection sampling, CoT traces
Real-world examples
Real products, models, and research that use this idea.
1DeepSeek distilled reasoning traces from DeepSeek-R1 into smaller Qwen and Llama students via SFT on generated chain-of-thought, an FT-based pipeline.
2Stanford Alpaca bootstrapped a Llama student by SFT on instruction-response pairs generated by a closed OpenAI teacher, a canonical FT-based distillation.
3Google's Gemma 2 9B and 2B were trained with logit-level distillation from a larger in-house teacher, feasible only because the teacher logits were accessible internally.
4Hugging Face TRL and the Unsloth toolkit ship reference SFT recipes that teams use to distill GPT-5.5 or Claude Opus 4.7 outputs into open-weight Llama 4 students.
5Together.ai and Fireworks publish distillation workflows where customers generate data from a frontier API teacher, rejection-sample it, then SFT a small hosted student.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does logit distillation transfer more than FT-based distillation, even on identical prompts?
Sign in or go Pro to see the approach for this follow-up.
QHow would you build an FT-based distillation pipeline for a reasoning student from a closed teacher?
Sign in or go Pro to see the approach for this follow-up.
QIf a closed API exposed top-20 logprobs, could you do partial logit distillation, and what breaks?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Claiming you can logit-distill from GPT-4o or Claude. Closed APIs expose at most top-k logprobs, never the full vocab softmax, so the KL loss cannot be computed. Only FT-based distillation works there.