Match each weight-quantization regime to what it buys you and where it breaks

Attempt it

FP8 (E4M3 / E5M2) weights

INT8 weights (W8)

INT4 weights (W4) via AWQ / GPTQ

Activation outliers at low-bit

Decode arithmetic intensity ~1

Quick reference

TL;DR

FP8 is near-lossless and native on new hardware, INT8 is a mature safe PTQ default, and INT4 buys 4x compression but needs AWQ or GPTQ to survive activation outliers.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine shipping a thick textbook by mail and paying by weight. FP8 prints it on slightly thinner paper, half the weight, and you can barely tell. INT8 uses cheaper but still readable paper, a well-tested trick that almost never goes wrong. INT4 shrinks the text to a quarter of the weight, which is great for postage, but now some fine print blurs and you need a careful reprinting process to keep it legible. The blurry parts are usually a few unusually important words that hate being squished. Special methods rescale those words first so they survive. Why care about weight at all? Because the mail cost here is reading numbers from memory, and lighter packages arrive faster.

Key concepts

Weight quantization FP8 precision AWQ and GPTQ Activation outliers

Quick scan · 2–4 min

Everything important, quickly.

4 min: place FP8, INT8, INT4 on the memory versus quality curve, explain activation outliers plus AWQ, GPTQ, SmoothQuant, then tie the win to decode being bandwidth-bound.

Side-by-side

Precision	Memory win	Quality	Calibration effort	Hardware note
FP8 (E4M3/E5M2)	~2x	Near FP16	Minimal	Native Hopper / Blackwell
INT8 (W8)	~2x	Near lossless	PTQ + per-channel	Broadly portable
INT4 (W4) AWQ/GPTQ	~4x	1-3 pt drop	Group-wise + outlier-aware	Best decode latency

Real-world examples

Real products, models, and research that use this idea.

NVIDIA TensorRT-LLM serves FP8 weights and activations natively on H100 and B200, the default minimal-effort path on Hopper and Blackwell.
vLLM and SGLang ship AWQ and GPTQ INT4 weight kernels so 70B-class models like Llama 4 fit on a single 80 GB GPU.
DeepSeek V4 and Qwen 3 publish official AWQ and GPTQ INT4 checkpoints for memory-constrained single-GPU serving.
SmoothQuant is the standard recipe for shifting activation outliers into weights before INT8 or INT4 post-training quantization.
Mistral Large 3 ships FP8 weights as the recommended production precision on current NVIDIA data-center hardware.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does INT4 need group-wise scales while INT8 can often use per-channel scales?

Sign in or go Pro to see the approach for this follow-up.

QHow does SmoothQuant differ from AWQ in handling activation outliers?

Sign in or go Pro to see the approach for this follow-up.

QWhy does weight quantization help decode far more than prefill?

Sign in or go Pro to see the approach for this follow-up.

QWhen would you quantize weights but keep the KV cache at higher precision?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.