Which statements about FP8, INT8 and INT4 weight quantization are correct?
Lower bit-width shrinks weight bytes near-proportionally, which speeds bandwidth-bound decode; but sub-8-bit needs calibration or salient-channel protection, and INT4 tensor-core support is generation-specific.
Imagine shipping a library by truck. The text is what matters, but the paper and ink are heavy, so the truck moves slowly. Shrinking the model is like reprinting every book on thinner paper with smaller type. FP8 is a modest trim that barely hurts readability. INT4 is aggressive shrinkage, so you must reprint carefully, keeping the important headings legible, or the books become unreadable. A lighter truck arrives faster, and since the trip was slowed by weight rather than reading speed, halving the cargo weight nearly halves the travel time. But not every depot owns the gear to handle the thinnest paper, so some have to puff the pages back up before anyone can read them.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Weight quantization stores model parameters in fewer bits than the FP16 baseline. It is one of the most effective inference optimizations in production. Low-bit weights are mainly a memory and bandwidth optimization, and the quality cost depends on how hard the format is to calibrate.
Three of the five statements are true. FP8 roughly halves weight bytes with near-zero quality loss on recent tensor cores. INT4 cuts them to a quarter but needs careful calibration methods to stay accurate. And because decode is bandwidth-bound, the latency gain scales nearly linearly with the bit reduction. Two statements are false: that INT8 is unsafe without quantization-aware training, and that sub-8-bit weights are free on every GPU generation.
Memory savings and latency savings are related but not identical. The storage format guarantees the memory savings. Latency savings depend on the workload being bandwidth-bound, which is true for small-batch decode but not for large-batch prefill or training. Much confusion comes from mixing up these two.
Why decode latency tracks weight bytes, not FLOPs
During autoregressive decode the model emits one token at a time. To produce that token, the GPU must read every weight matrix from high-bandwidth memory and multiply it by a single-token activation vector. The arithmetic intensity, meaning math operations per byte loaded, is close to one. The GPU finishes the math long before the next batch of weights arrives, so it sits idle waiting on memory.
This is what people mean by memory bandwidth bound. The dominant cost per token is streaming the weights, not the multiply-accumulate. So if you cut each weight from 16 bits to 4 bits, you move one quarter of the bytes per token, and decode time falls toward one quarter.
That is the core reason option C is true. The latency improvement is near-linear in the bit reduction, at least until software overhead, dequantization kernels, or batching shift the bottleneck. It also explains the most common conceptual error. People attribute the speedup to cheaper arithmetic, when in fact many low-bit paths still run the matmul in FP16.
The roofline model makes this concrete. A workload is bandwidth-bound whenever its arithmetic intensity is below the GPU's compute to bandwidth ratio, called the ridge point. An H100 SXM has about 989 dense FP16 TFLOPS and 3.35 TB/s of HBM bandwidth, so its ridge point is about 295 FLOPs per byte. Single-token decode, at about 1 FLOP per byte, is far below that ridge, deep in the memory-bound region. Each byte removed from the weights makes decode proportionally faster. So weight quantization gives the largest decode gain available, even before batching.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Format | Memory vs FP16 | Quality risk | Calibration need |
|---|---|---|---|
| FP8 (E4M3) | ~2x | Near-zero on Hopper/Blackwell | Minimal; exponent covers the range |
| INT8 (per-channel) | ~2x | Low; small drop | PTQ calibration set |
| INT4 (AWQ/GPTQ) | ~4x | Moderate; 1-3 point drop | Group scales plus salient protection |
Real products, models, and research that use this idea.
- vLLM and SGLang ship FP8 weight and KV cache support tuned for Hopper H100 and Blackwell B200, serving Llama 4 with near-FP16 quality.
- vLLM loads AWQ and GPTQ 4-bit checkpoints on Ampere and Hopper; at 4 bits the weights of Llama 3.1 70B take about 35 GB, so they fit on one 48 GB GPU.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does FP8 hold quality better than INT8 at the same bit count?
Compare the formats. FP8 spends bits on an exponent, giving it dynamic range across orders of magnitude. INT8 is uniform, so it must pick one scale per channel and clips or wastes resolution when values span a wide range. Discuss E4M3 versus E5M2 trade-offs.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming fewer bits always means proportionally faster matmul. The decode speedup comes from reading fewer weight bytes, not from cheaper arithmetic; many low-bit paths still compute in FP16.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.