Compare

Distillation vs Quantization

Two orthogonal ways to shrink a large language model

The verdict

Distillation shrinks the architecture (fewer parameters); quantization shrinks each parameter's bit count. They attack different axes of size and stack cleanly. QLoRA is quantization plus adapter fine-tuning.

Distillation

Glossary →

Train a smaller 'student' network to match the outputs (or intermediate states) of a bigger 'teacher'. The student has fewer parameters and runs faster, at some quality cost that depends on the gap and the distillation objective.

Best for: Making a fundamentally smaller model.

Quantization

Glossary →

Reduce the precision each parameter is stored in (FP16 to INT8 to INT4). The architecture is unchanged; each weight simply occupies fewer bits, shrinking memory and speeding up matrix multiplies on supporting hardware.

Best for: Fitting an existing model into tighter memory.

At a glance

Distillation vs Quantization: dimension-by-dimension comparison
Dimension	Distillation	Quantization
What shrinks	Parameter count	Bit width per parameter
Requires training	Yes (student network)	Not always (post-training methods exist)
Quality delta	Real gap vs teacher	8-bit ≈ FP16; below 4-bit becomes lossy
Memory savings	High (fewer weights)	High (4x from FP16 to 4-bit)
Common tools	DistilBERT, MiniLM, TinyLlama	GPTQ, AWQ, bitsandbytes, GGUF
Best for	Genuinely smaller model	Fitting large models into tight memory

Key differences

1Distillation changes the model's parameter count; quantization changes each parameter's bit width
2Distillation requires training a new network; quantization is post-training on the existing weights
3Quantization preserves the original architecture and most of the quality; distillation has a real quality gap versus the teacher
4Distillation output is a separate smaller checkpoint; quantized output is the same model with fewer bits per weight
5They compose: QLoRA quantizes a base model to 4-bit and trains LoRA adapters on top

In the interview

What they're really testing

Whether you know these are orthogonal compression axes and can name QLoRA as the canonical combine pattern.

Say this

Distillation trains a smaller student to imitate a teacher, so the resulting network has fewer parameters. Quantization keeps the same architecture but stores each weight in fewer bits. They are orthogonal, so they compose: QLoRA quantizes a base model to 4-bit and trains LoRA adapters on top, giving you the memory savings of both. In practice I reach for quantization first because it needs no retraining, and only distill when I genuinely need a smaller model.

Traps to sidestep

Claiming quantization always destroys quality
Calling distillation a form of quantization
Missing that QLoRA is a distillation-free combine
Forgetting that 8-bit is usually free while 3-bit and below is not

How to choose

If you need a genuinely smaller modelDistillation

If you need to fit an existing model in tighter memoryQuantization

If training budget is unavailableQuantization

If the student must serve a very different distributionDistillation

Fewer parameters → distill. Fewer bits per parameter → quantize. Both when maximal compression matters.

Common misconceptions

Myth: Quantization always hurts quality.

Reality: 8-bit is almost free for most modern LLMs. Quality only drops meaningfully below 4-bit, and even then AWQ and GPTQ recover most of it.

Myth: Distillation just copies the teacher.

Reality: It is an approximation with a real gap. The student never quite reaches the teacher's ceiling on hard examples.

Memory aid

Distillation is training a shorter apprentice; quantization is writing the same book in shorthand.

Can you combine them?

Yes, and it is common. Distill first to reduce the parameter count, then quantize to reduce the bit width. QLoRA is a lightweight version of the same idea: quantize the base, adapt with LoRA.

Related comparisons

API LLMvsSelf-hosted LLM

API vs Self-hosted LLMs

Rent the best model, or run your own

DDPvsFSDPvsZeRO

DDP vs FSDP vs ZeRO

Three ways to shard distributed LLM training across GPUs

DPOvsPPO

DPO vs PPO

Direct Preference Optimization vs Proximal Policy Optimization for RLHF

Greedy DecodingvsBeam SearchvsSampling (temperature, top-p, top-k)

Greedy vs Beam Search vs Sampling

Three ways to decode tokens from a language model

In-context learningvsFine-tuning

In-context learning vs Fine-tuning

Adapt in the prompt, or adapt in the weights

LoRA (Low-Rank Adaptation)vsFull Fine-tuning

LoRA vs Full Fine-tuning

Parameter-efficient adaptation vs updating all model weights