FT-based distillation vs logit distillation: and what works with closed APIs

Short answer·Medium·4.0 · 0·~3 min·Relevant atAnthropic Databricks Meta

Attempt it

Explain the difference between classical logit distillation and FT-based distillation. Which one works when your teacher is a closed-API model (GPT-4o, Claude), and why?

Free · 2 AI evals / day

**Knowledge distillation** trains a small student model to copy a bigger, smarter teacher. There are two common ways to do it, and they differ in what they copy. Logit distillation copies the teacher's full opinion. At each token the teacher produces a probability over the whole vocabulary, not just one winning word. The student is trained to match that whole distribution. This carries 'soft' information: how confident the teacher was, and which runner-up words it almost picked. The loss is KL divergence between the two distributions. FT-based distillation copies only the teacher's finished answers. The teacher generates text, and you do ordinary supervised fine-tuning on those (prompt, answer) pairs. The student never sees the soft distribution, only the words that actually came out. The loss is the usual next-token cross-entropy. Why does this matter with closed APIs? To run logit distillation you need the teacher's per-token probabilities. GPT-4o, Claude, and Gemini do not hand those over. So logit distillation is off the table for them. FT-based distillation only needs the teacher to write text, which every API does, so that is the method people actually use.

Both methods shrink a big teacher into a small student. They differ in the signal they extract. **Logit distillation** trains the student to match the teacher's full next-token distribution at every position. The loss is KL divergence between the two softmaxes over the vocabulary. This transfers soft knowledge: the teacher's confidence and the alternatives it nearly chose. Computing that loss requires the teacher's per-token logits or probabilities. FT-based distillation needs only generated text. The teacher writes responses, and you run standard supervised fine-tuning on the resulting prompt-response pairs. The student imitates the argmax behavior, not the soft distribution. The loss is plain next-token cross-entropy. Closed-API teachers settle the choice. Logit distillation needs the full vocab softmax. Closed APIs expose at most top-k logprobs, often capped or absent. So you cannot compute the KL term for a GPT-4o, Claude, or Gemini teacher. FT-based distillation works with any teacher you can query, since generation is all every API offers. You sample thousands to tens of thousands of pairs, then SFT the student. The signal is lossier; you discard the rejected-alternative information. It is the production standard for closed-API teachers. Teams usually add rejection sampling: generate several candidates, keep those passing a quality check, then SFT on the survivors.

Quick reference

TL;DR

Logit distillation matches the teacher's full softmax via KL and needs its logits. FT-based distillation just SFTs the student on teacher-generated text, so it works with any closed-API teacher.

Memory aid

Easy to grasp

Imagine learning chess from a grandmaster. The richest way is the master narrating not just the move they played, but how much they liked every other move too, a full ranked opinion on each turn. You copy that whole ranking. But some masters will only let you watch them play; they never share the rankings inside their head. So you fall back to the simpler way: watch thousands of their finished games and learn to imitate the moves you saw. You lose the rich 'how sure were they' signal, but you can learn from any master you can watch. To get cleaner lessons, you keep only their best games and study those.

Key concepts

Knowledge distillation KL divergence Rejection sampling Supervised fine-tuning

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: define both methods + the KL vs cross-entropy loss + what closed APIs expose + why FT-based is the only option there + rejection sampling and CoT traces + the licensing caveat.

Side-by-side

Aspect	Logit distillation	FT-based distillation
Signal copied	Full softmax over vocab (soft targets)	Sampled text only (hard targets)
Loss	KL divergence between distributions	Next-token cross-entropy (standard SFT)
Teacher access needed	Per-token logits plus aligned tokenizer	Text generation only
Works with closed APIs	No (full logits not exposed)	Yes (any callable teacher)
Information transferred	Confidence and rejected alternatives	Argmax behavior only (lossier)
Common augmentation	Temperature scaling on teacher	Rejection sampling, CoT traces

Real-world examples

Real products, models, and research that use this idea.

DeepSeek distilled reasoning traces from DeepSeek-R1 into smaller Qwen and Llama students via SFT on generated chain-of-thought, an FT-based pipeline.
Stanford Alpaca bootstrapped a Llama student by SFT on instruction-response pairs generated by a closed OpenAI teacher, a canonical FT-based distillation.
Google's Gemma 2 9B and 2B were trained with logit-level distillation from a larger in-house teacher, feasible only because the teacher logits were accessible internally.
Hugging Face TRL and the Unsloth toolkit ship reference SFT recipes that teams use to distill GPT-5.5 or Claude Opus 4.7 outputs into open-weight Llama 4 students.
Together.ai and Fireworks publish distillation workflows where customers generate data from a frontier API teacher, rejection-sample it, then SFT a small hosted student.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does logit distillation transfer more than FT-based distillation, even on identical prompts?

QHow would you build an FT-based distillation pipeline for a reasoning student from a closed teacher?

QIf a closed API exposed top-20 logprobs, could you do partial logit distillation, and what breaks?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Claiming you can logit-distill from GPT-4o or Claude. Closed APIs expose at most top-k logprobs, never the full vocab softmax, so the KL loss cannot be computed. Only FT-based distillation works there.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.