Closed APIs only return sampled text, so the feasible recipe is FT-based distillation: collect (prompt, teacher-response) pairs, then SFT the student with next-token cross-entropy.
Imagine learning to cook from a famous chef who will not share recipes, they only let you taste finished dishes. You cannot copy their exact measurements, the hidden numbers in their head. But you can taste enough plates and practice until your cooking matches. That is FT-based distillation. The locked teacher hands you finished answers, never its inner thoughts. So you collect lots of question and answer plates, then train your small student model to reproduce those answers. The fancier methods want the chef's private notes, the full probability numbers and the internal states, which a locked API simply never shares. So you settle for tasting and imitating. It is a lossier signal, but it always works with any teacher you can ask.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Knowledge distillation transfers capability from a large, expensive teacher model into a small, cheap student. The classic 2015 formulation trains the student to match the teacher's full output distribution, the soft probabilities over every vocabulary token, with a KL-divergence loss. Those soft targets carry more information than a one-hot label. They encode the teacher's relative confidence across alternatives, which is why distillation can be more sample-efficient than training the student from scratch on hard labels.
That textbook recipe quietly assumes white-box access to the teacher. You need the dense softmax at every step, and often the intermediate hidden states too. In 2026, the strongest teachers are frontier models served behind closed APIs, GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro. You send a prompt and you receive text. You may get top-k logprobs as a courtesy, but never the full vocabulary distribution, never the internal activations, never gradients.
This is the trap the question is built around. Three of the four options silently assume a level of access the API never grants. Only one method, distillation via fine-tuning, lives entirely within the text-only interface. This deep dive explains what each method actually requires, why the closed-API constraint eliminates the rest, and how to run the feasible recipe well.
It helps to fix the vocabulary first. Distillation signals come in three grades. Hard labels are the sampled completion text, the lowest-information signal but the only one a closed teacher gives up. Soft logits are the teacher's full probability distribution, a far richer signal that demands white-box access. Generated traces sit in between, full reasoning chains sampled as text, which preserve the teacher's step by step procedure even though each token is still just a hard label. The 2026 recipe leans on hard labels and traces, because soft logits are simply unavailable from the models worth distilling.
What a closed API actually exposes
Strip away the marketing and a chat completions API offers a narrow contract. You submit a prompt, optionally with sampling parameters, and you receive generated text. Many providers also return top-k logprobs, the log-probabilities of the few most likely tokens at each position, capped well below the full vocabulary size.
What you never receive is the dense softmax over the entire vocabulary, the model's internal hidden states during generation, or any gradient signal. These are precisely the quantities the classic distillation variants depend on. The provider has both a business and a safety incentive to keep them hidden. Exposing them would make the model trivial to clone.
So the design question becomes: which distillation method needs nothing beyond sampled text? That single constraint is the whole puzzle. Once you internalize it, the four options sort themselves immediately. Three demand hidden access, one does not.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Method | Signal needed | Closed-API feasible? |
|---|---|---|
| Logit / KL distillation | Full per-token softmax distribution | No: at most top-k logprobs exposed |
| Hidden-state / feature matching | Internal generation activations | No: embeddings endpoint differs |
| RL as reward (PPO) | Teacher scoring per rollout | Costly and brittle, not practical |
| FT-based distillation (SFT) | Sampled text completions only | Yes: the standard 2026 recipe |
Real products, models, and research that use this idea.
- Alpaca and Vicuna pioneered the recipe in 2023, fine-tuning Llama on completions sampled from a stronger closed teacher.
- Stanford's Alpaca used 52k self-instruct prompts answered by a closed OpenAI model, then SFT-distilled into a 7B Llama student.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf the API exposes top-k logprobs, can you do partial logit distillation?
Discuss matching a truncated, renormalized distribution over the top-k tokens. Note the bias from missing tail mass, and that most fine-tuning APIs do not accept soft targets anyway, so it rarely beats clean SFT in practice.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reaching for logit or hidden-state distillation against a closed API. Those need the full softmax or internal activations, which providers never expose. Only sampled text is available.
60 second bullets to scan on the way to the call.
What a closed API actually exposes versus withholds
Why logit KL distillation needs the full softmax
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.