Distillation with a closed-API teacher: which method is feasible?
Closed APIs only return sampled text, so the feasible recipe is FT-based distillation: collect (prompt, teacher-response) pairs, then SFT the student with next-token cross-entropy.
Imagine learning to cook from a famous chef who will not share recipes, they only let you taste finished dishes. You cannot copy their exact measurements, the hidden numbers in their head. But you can taste enough plates and practice until your cooking matches. That is FT-based distillation. The locked teacher hands you finished answers, never its inner thoughts. So you collect lots of question and answer plates, then train your small student model to reproduce those answers. The fancier methods want the chef's private notes, the full probability numbers and the internal states, which a locked API simply never shares. So you settle for tasting and imitating. It is a lossier signal, but it always works with any teacher you can ask.
Detailed answer & concept explanation~7 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: name the access constraint, give the sample-filter-SFT recipe, eliminate each distractor, then cover signal loss, data diversity, rejection sampling, and the terms of service risk.
| Method | Signal needed | Closed-API feasible? |
|---|---|---|
| Logit / KL distillation | Full per-token softmax distribution | No: at most top-k logprobs exposed |
| Hidden-state / feature matching | Internal generation activations | No: embeddings endpoint differs |
| RL as reward (PPO) | Teacher scoring per rollout | Costly and brittle, not practical |
| FT-based distillation (SFT) | Sampled text completions only | Yes: the standard 2026 recipe |
Real products, models, and research that use this idea.
- Alpaca and Vicuna pioneered the recipe in 2023, fine-tuning Llama on completions sampled from a stronger closed teacher.
- Stanford's Alpaca used 52k self-instruct prompts answered by a closed OpenAI model, then SFT-distilled into a 7B Llama student.
- OpenAI's distillation product pairs stored GPT-5.5 completions with the fine-tuning API to train cheaper small-tier students.
- DeepSeek's 2025 R1 release distilled long reasoning traces into smaller Qwen and Llama students, transferring chain-of-thought via SFT.
- Together.ai and Unsloth ship reference pipelines that sample teacher completions, rejection-sample them, then LoRA-fine-tune a Llama 4 student.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf the API exposes top-k logprobs, can you do partial logit distillation?
QHow do you stop the student from inheriting the teacher's hallucinations and errors?
QWhy distill chain-of-thought traces rather than only final answers for reasoning tasks?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Reaching for logit or hidden-state distillation against a closed API. Those need the full softmax or internal activations, which providers never expose. Only sampled text is available.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.