Glossary · Inference

Quantization

Also known as: INT8, INT4, GPTQ, AWQ

Run the model at lower numerical precision to save memory and accelerate inference.

Reducing the numerical precision of model weights (and sometimes activations) from FP16/FP32 to INT8, INT4, or lower. Cuts memory by 2-8x and often speeds up inference with minimal quality loss when done carefully.

In practice

The default lever for fitting big models onto smaller GPUs. Expect questions on activation-vs-weight quantization, GPTQ vs AWQ, and where quality starts to degrade.

How it compares

vs. Knowledge Distillation →Deep comparison →

Quantization shrinks each weight's bit count; distillation trains a smaller model from a bigger teacher.

Comparisons that include Quantization

Distillation vs QuantizationDistillation shrinks the architecture (fewer parameters); quantization shrinks each parameter's bit count. They attack different axes of size and stack cleanly. QLoRA is quantization plus adapter fine-tuning.

Related terms

KV Cache

Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.

Temperature

Sampling knob (low = focused, high = diverse), applied to the logits before softmax.

FlashAttention

A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.

Grouped-Query Attention (GQA)

Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.

Knowledge Distillation

Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.

vLLM

An open-source serving engine with PagedAttention. Much higher throughput than naive HF Transformers serving.