Match each weight-quantization regime to what it buys you and where it breaks
FP8 is near-lossless and native on new hardware, INT8 is a mature safe PTQ default, and INT4 buys 4x compression but needs AWQ or GPTQ to survive activation outliers.
Imagine shipping a thick textbook by mail and paying by weight. FP8 prints it on slightly thinner paper, half the weight, and you can barely tell. INT8 uses cheaper but still readable paper, a well-tested trick that almost never goes wrong. INT4 shrinks the text to a quarter of the weight, which is great for postage, but now some fine print blurs and you need a careful reprinting process to keep it legible. The blurry parts are usually a few unusually important words that hate being squished. Special methods rescale those words first so they survive. Why care about weight at all? Because the mail cost here is reading numbers from memory, and lighter packages arrive faster.
Detailed answer & concept explanation~7 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: place FP8, INT8, INT4 on the memory versus quality curve, explain activation outliers plus AWQ, GPTQ, SmoothQuant, then tie the win to decode being bandwidth-bound.
| Precision | Memory win | Quality | Calibration effort | Hardware note |
|---|---|---|---|---|
| FP8 (E4M3/E5M2) | ~2x | Near FP16 | Minimal | Native Hopper / Blackwell |
| INT8 (W8) | ~2x | Near lossless | PTQ + per-channel | Broadly portable |
| INT4 (W4) AWQ/GPTQ | ~4x | 1-3 pt drop | Group-wise + outlier-aware | Best decode latency |
Real products, models, and research that use this idea.
- NVIDIA TensorRT-LLM serves FP8 weights and activations natively on H100 and B200, the default minimal-effort path on Hopper and Blackwell.
- vLLM and SGLang ship AWQ and GPTQ INT4 weight kernels so 70B-class models like Llama 4 fit on a single 80 GB GPU.
- DeepSeek V4 and Qwen 3 publish official AWQ and GPTQ INT4 checkpoints for memory-constrained single-GPU serving.
- SmoothQuant is the standard recipe for shifting activation outliers into weights before INT8 or INT4 post-training quantization.
- Mistral Large 3 ships FP8 weights as the recommended production precision on current NVIDIA data-center hardware.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does INT4 need group-wise scales while INT8 can often use per-channel scales?
QHow does SmoothQuant differ from AWQ in handling activation outliers?
QWhy does weight quantization help decode far more than prefill?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating all three precisions as interchangeable knobs. FP8, INT8, and INT4 sit at different points on the memory versus quality curve, and only INT4 routinely needs outlier-aware calibration.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
- Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression
- Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Xiao et al., SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs
- Micikevicius et al., FP8 Formats for Deep Learning
Same topic, related formats. Practice these next.