Match each weight quantization regime to what it buys you and where it breaks
Drag each answer to line up with its matching prompt
FP8 (E4M3 / E5M2) weights
Where INT4 typically breaks; mitigated by SmoothQuant / AWQ scaling
INT8 weights (W8)
~FP16 quality, 2× memory + bandwidth win, native on Hopper/Blackwell: minimal effort acceleration
INT4 weights (W4) via AWQ / GPTQ
Why smaller weights cut decode time directly (bandwidth bound regime)
Activation outliers at low bit
4× memory win, ~1-3 pt quality drop, needs group wise scales: biggest decode latency improvement
Decode arithmetic intensity ~1
Broadly safe with PTQ + per channel calibration; ~free on most architectures
FP8 is near-lossless and native on new hardware, INT8 is a mature safe PTQ default, and INT4 gives 4x compression but needs AWQ or GPTQ to keep accuracy despite activation outliers.
Imagine shipping a thick textbook by mail and paying by weight. FP8 prints it on slightly thinner paper, half the weight, and you can barely tell. INT8 uses cheaper but still readable paper, a well-tested trick that almost never goes wrong. INT4 shrinks the text to a quarter of the weight, which is great for postage, but now some fine print blurs and you need a careful reprinting process to keep it legible. The blurry parts are usually a few unusually important words that lose the most when shrunk. Special methods rescale those words first so they survive. Why care about weight at all? Because the mail cost here is reading numbers from memory, and lighter packages arrive faster.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Weight quantization is the practice of storing a model's parameters in fewer bits than the 16-bit float used during training. The payoff is twofold. The model occupies less memory, and during inference there are fewer bytes to stream from high-bandwidth memory per token. Because the decode phase of LLM serving is bound by memory bandwidth rather than raw compute, fewer bytes per weight read translates almost directly into faster token generation.
FP8, INT8, and INT4 are not three settings of one control. Each sits at a distinct point on a memory versus quality curve, each has a different relationship to the hardware, and only the most aggressive of them routinely needs special calibration to stay accurate.
One number anchors the whole picture: the bytes per weight. FP16 spends two. FP8 and INT8 spend one. INT4 spends roughly a half plus a little overhead for scales. That number sets both the memory footprint and, on decode, the latency. The rest is about how far you can push the byte count down before quality drops, and which methods let you go further.
FP8: near-lossless and hardware-native
FP8 is an 8-bit floating-point format with two common variants. E4M3 uses four exponent bits and three mantissa bits, favoring precision, and is the usual choice for weights. E5M2 uses five exponent bits and two mantissa bits, favoring dynamic range, and shows up where wide ranges matter such as gradients.
The defining property of FP8 is that it keeps quality close to FP16 with very little effort. Because it is a float, its values are spaced more densely near zero, unlike the uniform spacing of an integer grid. That makes it forgiving for the heavy-tailed magnitude distributions found in transformer weights.
The second defining property is hardware. NVIDIA Hopper and Blackwell tensor cores execute FP8 matrix multiplies natively, and on an H100 SXM the dense FP8 rate (about 1,979 TFLOPS) is twice the dense BF16 rate. On those chips FP8 gives about a 2x saving in both memory footprint and bandwidth, with no integer dequantization step in the critical path. That is why it is the lowest-effort speedup on that hardware. It loses much of its appeal on older hardware without FP8 tensor cores, such as the A100.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Precision | Memory win | Quality | Calibration effort | Hardware note |
|---|---|---|---|---|
| FP8 (E4M3/E5M2) | ~2x | Near FP16 | Minimal | Native Hopper / Blackwell |
| INT8 (W8) | ~2x | Near lossless | PTQ + per-channel | Broadly portable |
| INT4 (W4) AWQ/GPTQ | ~4x | 1-3 pt drop | Group-wise + outlier-aware | Best decode latency |
Real products, models, and research that use this idea.
- NVIDIA TensorRT-LLM serves FP8 weights and activations natively on H100 and B200, the lowest-effort speedup path on Hopper and Blackwell.
- vLLM and SGLang ship AWQ and GPTQ INT4 weight kernels, so a 70B model such as Llama 3.1 70B (about 35 to 40 GB in INT4) fits on a single 80 GB GPU.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does INT4 need group-wise scales while INT8 can often use per-channel scales?
Think about how much dynamic range each scale must cover. At 4 bits there are only 16 levels, so one scale per channel cannot track local magnitude variation. Smaller groups, size 128, give each block its own scale and recover accuracy.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating all three precisions as interchangeable settings. FP8, INT8, and INT4 sit at different points on the memory versus quality curve, and only INT4 routinely needs outlier-aware calibration.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
- Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression
- Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Xiao et al., SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs
- Micikevicius et al., FP8 Formats for Deep Learning
Same topic, related formats. Practice these next.