Which statements about FP8, INT8 and INT4 weight quantization are correct?
Lower bit-width shrinks weight bytes near-proportionally, which speeds bandwidth-bound decode; but sub-8-bit needs calibration or salient-channel protection, and INT4 tensor-core support is generation-specific.
Imagine shipping a library by truck. The text is what matters, but the paper and ink are heavy, so the truck moves slowly. Shrinking the model is like reprinting every book on thinner paper with smaller type. FP8 is a modest trim that barely hurts readability. INT4 is aggressive shrinkage, so you must reprint carefully, keeping the important headings legible, or the books become unreadable. A lighter truck arrives faster, and since the trip was slowed by weight rather than reading speed, halving the cargo weight nearly halves the travel time. But not every depot owns the gear to handle the thinnest paper, so some have to puff the pages back up before anyone can read them.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Weight quantization is the practice of storing model parameters in fewer bits than the FP16 baseline, and it is one of the highest-leverage inference optimizations in production. The question lists five claims, three true and two false, and getting them right requires a single mental model: low-bit weights are primarily a memory and bandwidth optimization, and the quality cost depends on how hard the format is to calibrate.
The three true statements describe the consensus production picture. FP8 roughly halves weight bytes with near-zero quality loss on recent tensor cores. INT4 quarters them but needs calibration-heavy methods to stay accurate. And because decode is bandwidth-bound, the latency win scales nearly linearly with the bit reduction. The two false statements are classic traps: that INT8 is unsafe without quantization-aware training, and that INT4 runs natively on every GPU generation.
This deep dive builds the model from the ground up. It explains why decode is bandwidth-bound, how memory savings translate into latency, why FP8 and INT formats behave differently on quality, what AWQ and GPTQ actually do, and where hardware support and dequantization overhead break the simple story.
Keep one distinction in mind throughout. Memory savings and latency savings are related but not identical. Memory savings are guaranteed by the storage format. Latency savings depend on the workload being bandwidth-bound, which is true for small-batch decode but not for large-batch prefill or training. Most of the confusion in interviews comes from blurring these two, so the deep dive keeps them separate at every step.
Why decode latency tracks weight bytes, not FLOPs
During autoregressive decode the model emits one token at a time. To produce that token, the GPU must read every weight matrix from high-bandwidth memory and multiply it by a single-token activation vector. The arithmetic intensity, meaning math operations per byte loaded, is close to one. The GPU finishes the math long before the next batch of weights arrives, so it sits idle waiting on memory.
This is what people mean by memory bandwidth bound. The dominant cost per token is streaming the weights, not the multiply-accumulate. So if you cut each weight from 16 bits to 4 bits, you move one quarter of the bytes per token, and decode time falls toward one quarter.
That is the core reason option C is true. The latency improvement is near-linear in the bit reduction, at least until software overhead, dequantization kernels, or batching shift the bottleneck. It also explains the most common conceptual error. People attribute the speedup to cheaper arithmetic, when in fact many low-bit paths still run the matmul in FP16.
The roofline model makes this concrete. A workload is bandwidth-bound whenever its arithmetic intensity sits below the GPU's compute to bandwidth ratio, sometimes called the ridge point. An H100 has roughly 989 FP16 TFLOPs and about 3.35 TB/s of HBM bandwidth, so its ridge point is in the hundreds of FLOPs per byte. Single-token decode lives far to the left of that ridge, deep in the memory-bound region. Every byte you remove from the weights moves you proportionally faster, which is precisely why weight quantization is the single highest-leverage decode lever before you even touch batching.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Format | Memory vs FP16 | Quality risk | Calibration need |
|---|---|---|---|
| FP8 (E4M3) | ~2x | Near-zero on Hopper/Blackwell | Minimal; exponent absorbs range |
| INT8 (per-channel) | ~2x | Low; small drop | PTQ calibration set |
| INT4 (AWQ/GPTQ) | ~4x | Moderate; 1-3 point drop | Group scales plus salient protection |
Real products, models, and research that use this idea.
- vLLM and SGLang ship FP8 weight and KV cache support tuned for Hopper H100 and Blackwell B200, serving Llama 4 with near-FP16 quality.
- AWQ and GPTQ are the default 4-bit recipes in Hugging Face Transformers and llama.cpp, used to fit Llama 3.1 70B on a single 48 GB GPU.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does FP8 hold quality better than INT8 at the same bit count?
Compare the formats. FP8 spends bits on an exponent, giving it dynamic range across orders of magnitude. INT8 is uniform, so it must pick one scale per channel and clips or wastes resolution when values span a wide range. Discuss E4M3 versus E5M2 trade-offs.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming fewer bits always means proportionally faster matmul. The decode speedup comes from reading fewer weight bytes, not from cheaper arithmetic; many low-bit paths still compute in FP16.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.