Quantization
Also known as: INT8, INT4, GPTQ, AWQ
Run the model at lower numerical precision to save memory and accelerate inference.
Reducing the numerical precision of model weights (and sometimes activations) from FP16/FP32 to INT8, INT4, or lower. Cuts memory by 2-8x and often speeds up inference with minimal quality loss when done carefully.
In practice
The default lever for fitting big models onto smaller GPUs. Expect questions on activation-vs-weight quantization, GPTQ vs AWQ, and where quality starts to degrade.
How it compares
Quantization shrinks each weight's bit count; distillation trains a smaller model from a bigger teacher.
Comparisons that include Quantization
Related topics
Questions that mention this term
- INT8 KV cache vs INT8 weight quantization, which one is easier in production?
- When does binary quantization…
- Misconception: 'LoRA is just…
- Pick the most accurate characterization of how TGI compares with vLLM in 2026
- Which statements about QLoRA are true?
- Retrieval P95 latency tripled after the corpus grew from 1M to 20M vectors: diagnose it
Related terms
API LLM
An LLM accessed through a provider API: pay per token, get the frontier model, hand over ops.
Beam Search
Keep the K best partial sequences at each step; deterministic, breadth-first decoding.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
GGUF
Self-contained binary format for quantized LLMs; the standard for llama.cpp / Ollama / LM Studio.
Greedy Decoding
At each step, pick the single highest-probability token. Fast and deterministic, but often loops.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.