**Distillation** means training a small student model to copy a big teacher model. The goal is to keep most of the teacher's quality at a fraction of the cost and latency.
Classic distillation matches the teacher's full output probabilities, the entire softmax over every vocabulary token, with a KL loss. That needs deep access to the teacher's internals.
A closed API like the one serving GPT-4o gives you none of that. You send a prompt, you get text back. No full probability distribution, no hidden states, no gradients. So the textbook logit-matching recipe is simply impossible here.
The method that works is FT-based distillation. You query the teacher for thousands of completions, build a dataset of (prompt, teacher-response) pairs, then supervised-fine-tune the student on those pairs with ordinary next-token cross-entropy. The student learns to imitate the teacher's text.
It is a lossier signal than logit matching, you only see the sampled answer, not the probabilities behind it. But it works with any teacher you can call, which is why it is the common 2026 recipe.
The answer is FT-based distillation. The constraint drives everything. A closed API exposes only sampled text. It does not expose the full softmax, the hidden states, or the gradients the fancier methods need.
So you query the teacher for completions, assemble (prompt, response) pairs, then SFT the student with standard next-token cross-entropy. The student imitates the teacher's surface behavior. This is lossier than logit distillation, you lose the soft-probability signal, but it works with any teacher you can call.
Why the others fail is the real test. Full-softmax KL distillation needs the teacher's complete per-token distribution. The API caps you at top-k logprobs at best. Hidden-state matching needs internal activations the embeddings endpoint never returns. Using the teacher as a PPO reward model means an API call per sampled rollout. That is brittle and burns budget for worse results.
The practical recipe in 2026 is to generate diverse prompts, sample teacher completions, filter by rejection sampling, then LoRA-fine-tune an open student like Llama 4. Capability transfers cheaply this way.
Quick reference
TL;DR
Closed APIs only return sampled text, so the feasible recipe is FT-based distillation: collect (prompt, teacher-response) pairs, then SFT the student with next-token cross-entropy.
Imagine learning to cook from a famous chef who will not share recipes, they only let you taste finished dishes. You cannot copy their exact measurements, the hidden numbers in their head. But you can taste enough plates and practice until your cooking matches. That is FT-based distillation. The locked teacher hands you finished answers, never its inner thoughts. So you collect lots of question and answer plates, then train your small student model to reproduce those answers. The fancier methods want the chef's private notes, the full probability numbers and the internal states, which a locked API simply never shares. So you settle for tasting and imitating. It is a lossier signal, but it always works with any teacher you can ask.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You have full white-box access to the teacher's weights and logits Then true logit or hidden-state distillation gives a richer signal than text-only SFT, so use it.
The provider's terms of service forbid training models on their outputs Distilling a closed competitor's API can breach contract and create legal and compliance exposure.
The task needs fresh or private facts the teacher never saw Distillation copies behavior, not live knowledge. Retrieval augmentation handles freshness, not SFT on completions.
You cannot collect a diverse, on-distribution prompt set Narrow or skewed prompts make the student overfit a thin slice and miss the teacher's real coverage.
Quick scan · 2–4 min
Everything important, quickly.
4 min: name the access constraint, give the sample-filter-SFT recipe, eliminate each distractor, then cover signal loss, data diversity, rejection sampling, and the terms of service risk.
Side-by-side
Method
Signal needed
Closed-API feasible?
Logit / KL distillation
Full per-token softmax distribution
No: at most top-k logprobs exposed
Hidden-state / feature matching
Internal generation activations
No: embeddings endpoint differs
RL as reward (PPO)
Teacher scoring per rollout
Costly and brittle, not practical
FT-based distillation (SFT)
Sampled text completions only
Yes: the standard 2026 recipe
Real-world examples
Real products, models, and research that use this idea.
1Alpaca and Vicuna pioneered the recipe in 2023, fine-tuning Llama on completions sampled from a stronger closed teacher.
2Stanford's Alpaca used 52k self-instruct prompts answered by a closed OpenAI model, then SFT-distilled into a 7B Llama student.
3OpenAI's distillation product pairs stored GPT-5.5 completions with the fine-tuning API to train cheaper small-tier students.
4DeepSeek's 2025 R1 release distilled long reasoning traces into smaller Qwen and Llama students, transferring chain-of-thought via SFT.
5Together.ai and Unsloth ship reference pipelines that sample teacher completions, rejection-sample them, then LoRA-fine-tune a Llama 4 student.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf the API exposes top-k logprobs, can you do partial logit distillation?
Sign in or go Pro to see the approach for this follow-up.
QHow do you stop the student from inheriting the teacher's hallucinations and errors?
Sign in or go Pro to see the approach for this follow-up.
QWhy distill chain-of-thought traces rather than only final answers for reasoning tasks?
Sign in or go Pro to see the approach for this follow-up.
QHow does data diversity trade off against sheer volume of teacher completions?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reaching for logit or hidden-state distillation against a closed API. Those need the full softmax or internal activations, which providers never expose. Only sampled text is available.