FT based distillation vs logit distillation: and what works with closed APIs

Short answer·Medium·4.0 · 0·~3 min·Asked atRunway Sap Sharechat·Relevant atAnthropic Databricks Meta

Attempt it

Explain the difference between classical logit distillation and FT based distillation. Which one works when your teacher is a closed API model (GPT-4o, Claude), and why?

Free · 2 AI evals / day

**Knowledge distillation** trains a small student model to copy a bigger, smarter teacher. There are two common ways to do it, and they differ in what they copy. Logit distillation copies the teacher's full opinion. At each token the teacher produces a probability over the whole vocabulary, not just one winning word. The student is trained to match that whole distribution. This carries 'soft' information: how confident the teacher was, and which runner-up words it almost picked. The loss is KL divergence between the two distributions. FT-based distillation copies only the teacher's finished answers. The teacher generates text, and you do ordinary supervised fine-tuning on those (prompt, answer) pairs. The student never sees the soft distribution, only the words that actually came out. The loss is the usual next-token cross-entropy. Why does this matter with closed APIs? To run logit distillation you need the teacher's per-token probabilities. GPT-4o, Claude, and Gemini do not hand those over. So logit distillation is off the table for them. FT-based distillation only needs the teacher to write text, which every API does, so that is the method people actually use.

Both methods shrink a big teacher into a small student. They differ in the signal they extract. **Logit distillation** trains the student to match the teacher's full next-token distribution at every position. The loss is KL divergence between the two softmaxes over the vocabulary. This transfers soft knowledge: the teacher's confidence and the alternatives it nearly chose. Computing that loss requires the teacher's per-token logits or probabilities. FT-based distillation needs only generated text. The teacher writes responses, and you run standard supervised fine-tuning on the resulting prompt-response pairs. The student imitates the argmax behavior, not the soft distribution. The loss is plain next-token cross-entropy. Closed-API teachers settle the choice. Logit distillation needs the full vocab softmax. Closed APIs expose at most top-k logprobs, often capped or absent. So you cannot compute the KL term for a GPT-4o, Claude, or Gemini teacher. FT-based distillation works with any teacher you can query, since generation is all every API offers. You sample thousands to tens of thousands of pairs, then SFT the student. The signal is lossier; you discard the rejected-alternative information. It is the production standard for closed-API teachers. Teams usually add rejection sampling: generate several candidates, keep those passing a quality check, then SFT on the survivors.

TL;DR

Logit distillation matches the teacher's full softmax via KL and needs its logits. FT-based distillation just SFTs the student on teacher-generated text, so it works with any closed-API teacher.

Memory aid

Easy to grasp

Imagine learning chess from a grandmaster. The richest way is the master narrating not just the move they played, but how much they liked every other move too, a full ranked opinion on each turn. You copy that whole ranking. But some masters will only let you watch them play; they never share the rankings inside their head. So you fall back to the simpler way: watch thousands of their finished games and learn to imitate the moves you saw. You lose the rich 'how sure were they' signal, but you can learn from any master you can watch. To get cleaner lessons, you keep only their best games and study those.

Key concepts

Knowledge distillation KL divergence Rejection sampling Supervised fine-tuning

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Knowledge distillation answers a practical question: you have a big, capable teacher model and you want a small, cheap student that behaves almost as well. The mechanism that transfers capability is where the two methods diverge, and that divergence decides which one you can actually run.

The interview trap is to describe distillation generically and forget that the loss function dictates what access you need to the teacher. Classical logit distillation reaches inside the teacher and copies its full probability distribution over the vocabulary at every step. FT-based distillation never reaches inside; it only watches the text the teacher produces and trains the student to reproduce it. One demands white-box access. The other needs nothing but an API call.

That distinction is the whole answer when the teacher is a closed model like GPT-4o, Claude, or Gemini. You cannot see those models' logits, so you cannot compute the distillation KL loss against them. You can, however, ask them to generate as much text as your budget allows. This deep dive walks both mechanisms, the closed-API constraint, the data craft that makes FT-based distillation work, and the comparison to training a small model from scratch.

Logit distillation: matching the soft distribution

In logit distillation the student learns to reproduce the teacher's entire next-token probability distribution, not just the winning token. At each position the teacher emits a softmax over the whole vocabulary, and the student is trained to make its own softmax match it. The objective is KL divergence between the two distributions.

The canonical loss, with temperature scaling to soften both distributions, is:

D_{KL}(P_T \,\Vert\, P_S) = \sum_i P_T(i) \log \frac{P_T(i)}{P_S(i)}

The payoff is dark knowledge: the relative probability mass the teacher placed on tokens it did not pick. That signal tells the student which alternatives were close, which were absurd, and how confident the teacher was. A one-hot label can never carry that. This is why a well-distilled small model can outperform the same architecture trained on hard labels alone.

The cost is access. You need the teacher's per-token logits, a tokenizer that lines up position for position with the student, and ideally control over the teacher's sampling temperature. The temperature matters because Hinton's framing softens both distributions by the same factor, spreading probability mass so the student sees the structure among the non-top tokens rather than a near one-hot spike. All three requirements are realistic only when you host the teacher yourself, which in practice means an open-weight teacher.

There is also a subtler benefit worth naming in an interview. Because the gradient flows from a dense target rather than a single label, each training token is far more informative, so logit distillation typically converges with fewer examples than FT-based distillation reaches the same quality on. When you genuinely control both models, it is the better method. The reason it loses in practice is purely about access, not about the learning signal.

FT-based distillation: SFT on teacher text

Why closed APIs force FT-based distillation

Data craft: rejection sampling and reasoning traces

Distillation vs training from scratch

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Aspect	Logit distillation	FT-based distillation
Signal copied	Full softmax over vocab (soft targets)	Sampled text only (hard targets)
Loss	KL divergence between distributions	Next-token cross-entropy (standard SFT)
Teacher access needed	Per-token logits plus aligned tokenizer	Text generation only
Works with closed APIs	No (full logits not exposed)	Yes (any callable teacher)
Information transferred	Confidence and rejected alternatives	Argmax behavior only (lossier)
Common augmentation	Temperature scaling on teacher	Rejection sampling, CoT traces

Real products, models, and research that use this idea.

DeepSeek distilled reasoning traces from DeepSeek-R1 into smaller Qwen and Llama students via SFT on generated chain-of-thought, an FT-based pipeline.
Stanford Alpaca bootstrapped a Llama student by SFT on instruction-response pairs generated by a closed OpenAI teacher, a canonical FT-based distillation.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does logit distillation transfer more than FT-based distillation, even on identical prompts?

Compare the targets. Soft KL targets carry mass on rejected tokens (dark knowledge), giving a denser gradient. Cross-entropy on sampled text is a one-hot target that discards the teacher's confidence and alternatives.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Claiming you can logit-distill from GPT-4o or Claude. Closed APIs expose at most top-k logprobs, never the full vocab softmax, so the KL loss cannot be computed. Only FT-based distillation works there.

60 second bullets to scan on the way to the call.

What signal logit distillation copies versus FT-based
Why the KL loss needs the full vocabulary softmax

Primary sources. Browse if you want the original framing.