Glossary · Training

Knowledge Distillation

Also known as: Model distillation, Teacher-student training

Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.

Training a smaller 'student' model to mimic the behavior of a larger 'teacher' model, typically by matching the teacher's soft probability distributions or hidden representations. Produces faster, cheaper models with most of the teacher's quality.

In practice

How most production small models (Gemma, Phi, Mistral-7B-class) are made. Knowing it pairs naturally with quantization questions.

How it compares

vs. Quantization →Deep comparison →

Distillation shrinks the architecture (fewer params); quantization shrinks each param's bit count.

Comparisons that include Knowledge Distillation

Distillation vs QuantizationDistillation shrinks the architecture (fewer parameters); quantization shrinks each parameter's bit count. They attack different axes of size and stack cleanly. QLoRA is quantization plus adapter fine-tuning.

Related terms

Fine-tuning

Continue training a pre-trained model on task-specific data to specialize its behavior.

KV Cache

Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.

LoRA (Low-Rank Adaptation)

Fine-tune by training small low-rank update matrices instead of every weight: same quality, ~1% of the cost.

Temperature

Sampling knob (low = focused, high = diverse), applied to the logits before softmax.

Quantization

Run the model at lower numerical precision to save memory and accelerate inference.

FlashAttention

A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.