Concepts · Training

Knowledge Distillation

Also known as: Model distillation, Teacher-student training

Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.

Training a smaller 'student' model to mimic the behavior of a larger 'teacher' model, typically by matching the teacher's soft probability distributions or hidden representations. Produces faster, cheaper models with most of the teacher's quality.

In practice

How most production small models (Gemma, Phi, Mistral-7B-class) are made. Knowing it pairs naturally with quantization questions.

How it compares

vs. Quantization →Deep comparison →

Distillation shrinks the architecture (fewer params); quantization shrinks each param's bit count.

Comparisons that include Knowledge Distillation

Distillation vs QuantizationDistillation shrinks the architecture (fewer parameters); quantization shrinks each parameter's bit count. They attack different axes of size and stack cleanly. QLoRA is quantization plus adapter fine-tuning.

Questions that mention this term

Related terms

API LLM

An LLM accessed through a provider API: pay per token, get the frontier model, hand over ops.

Beam Search

Keep the K best partial sequences at each step; deterministic, breadth-first decoding.

DPO (Direct Preference Optimization)

Skip the reward model and PPO: fine-tune directly on preferred-vs-rejected response pairs.

Fine-tuning

Continue training a pre-trained model on task-specific data to specialize its behavior.

FlashAttention

A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.

FSDP (Fully Sharded Data Parallel)

Shard weights+grads+optimizer states across GPUs and gather just-in-time; train huge models with limited per-GPU memory.