Which statements about FP8, INT8 and INT4 weight quantization are correct?

Attempt it

Quick reference

TL;DR

Lower bit-width shrinks weight bytes near-proportionally, which speeds bandwidth-bound decode; but sub-8-bit needs calibration or salient-channel protection, and INT4 tensor-core support is generation-specific.

Memory aid

Easy to grasp

Imagine shipping a library by truck. The text is what matters, but the paper and ink are heavy, so the truck moves slowly. Shrinking the model is like reprinting every book on thinner paper with smaller type. FP8 is a modest trim that barely hurts readability. INT4 is aggressive shrinkage, so you must reprint carefully, keeping the important headings legible, or the books become unreadable. A lighter truck arrives faster, and since the trip was slowed by weight rather than reading speed, halving the cargo weight nearly halves the travel time. But not every depot owns the gear to handle the thinnest paper, so some have to puff the pages back up before anyone can read them.

Key concepts

Weight quantizationBandwidth-bound decodeFP8 AWQ and GPTQ

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: bandwidth-bound decode, then memory savings scale with bits, then FP8 keeps range, then INT4 needs AWQ or GPTQ, then hardware support is generation-specific, then W4A16 dequantize overhead.

Side-by-side

Format	Memory vs FP16	Quality risk	Calibration need
FP8 (E4M3)	~2x	Near-zero on Hopper/Blackwell	Minimal; exponent absorbs range
INT8 (per-channel)	~2x	Low; small drop	PTQ calibration set
INT4 (AWQ/GPTQ)	~4x	Moderate; 1-3 point drop	Group scales plus salient protection

Real-world examples

Real products, models, and research that use this idea.

vLLM and SGLang ship FP8 weight and KV cache support tuned for Hopper H100 and Blackwell B200, serving Llama 4 with near-FP16 quality.
AWQ and GPTQ are the default 4-bit recipes in Hugging Face Transformers and llama.cpp, used to fit Llama 3.1 70B on a single 48 GB GPU.
NVIDIA TensorRT-LLM exposes FP8 and INT4 W4A16 paths, with native FP8 tensor cores on Hopper and Blackwell but dequantized INT4 on older Ampere A100.
DeepSeek V4 and Mistral Large 3 publish FP8 checkpoints so serving stacks can skip a calibration step entirely.
Microsoft's BitNet research pushes toward ternary and sub-2-bit weights, showing the calibration burden grows sharply below 4 bits.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does FP8 hold quality better than INT8 at the same bit count?

QIf decode is bandwidth-bound, why does INT4 sometimes not hit a full 4x speedup?

QWhat makes the sub-8-bit regime specifically about outliers?

QHow does activation quantization change the picture versus weight-only?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming fewer bits always means proportionally faster matmul. The decode speedup comes from reading fewer weight bytes, not from cheaper arithmetic; many low-bit paths still compute in FP16.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.