Weight quantization stores model weights in a lower bit-width (FP8, INT8, INT4) than the FP16 baseline so each decode step streams fewer bytes from HBM; activations usually stay in higher precision and the matmul
Imagine a giant library where every book is written in a large, fancy font that takes up lots of shelf space. Picture the same library rewritten in a smaller font. The story is the same, the words mean the same thing, but each book is thinner so you can grab one off the shelf faster. The reader (the GPU) still understands the smaller font fine. The lights and chairs in the library (the running-around numbers and the actual reading) stay normal-sized. The whole point is that fetching books from the shelf was the slowest step, so making the books thinner makes the whole library run faster. The technique that compresses a model's learned numbers this way is what people mean by weight compression.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Weight quantization is one of the highest-leverage levers in production inference, and it is also one of the most casually mis-described in interviews. People hear 'quantization' and reach for 'smaller model', when the precise picture is much more interesting and explains both why the technique works so well and where it can fail.
The central distinction to fix in your head: a deployed transformer holds two very different kinds of numbers. Weights are the multi-gigabyte tensors trained once and read on every forward pass. Activations are the much smaller tensors that flow through the model at runtime and are thrown away each step. Standard weight quantization shrinks the first group and leaves the second largely alone.
That asymmetry is not arbitrary. Decode is memory-bandwidth bound: the speed limit is how fast the GPU can stream weights from HBM, not how fast it can multiply numbers. Shrinking the bytes per weight directly attacks that bottleneck. Activations, which are not the bottleneck, can stay in FP16 / BF16 where they are easy to handle and quality is fragile.
This deep dive walks through the bit-width ladder, what happens at the matmul, how modern hardware changed the calculus, and what the realistic quality cost looks like in 2026.
The bit-width ladder: from FP16 to FP4
The starting point is FP16 or BF16: 2 bytes per weight. This is the precision a model is typically trained in, and on Hopper / Blackwell it is fully supported in tensor cores, so it is the no-tradeoff baseline.
FP8 (E4M3 or E5M2) sits one rung down. 1 byte per weight, exactly half the HBM footprint and half the streaming bandwidth. H100 introduced native FP8 tensor cores, so the matmul itself can run in FP8 once the weight is loaded. Post-training calibration (a calibration pass over a small dataset to set per-tensor scaling factors) recovers essentially all accuracy on chat workloads.
INT8 is the integer cousin of FP8: same 1 byte per weight, same 2x bandwidth and HBM saving. The difference is that the matmul runs in integer math via INT8 tensor cores. INT8 paths are mature on every modern accelerator and are common in serving stacks that pre-date wide FP8 support.
INT4 (W4A16, via GPTQ or AWQ) is the aggressive consumer-friendly tier. 0.5 bytes per weight gives a 4x bandwidth and HBM saving. Activations stay in FP16. The kernel reads a packed INT4 block, dequantizes it on the fly to FP16, and multiplies against the FP16 activation. The compute does not get cheaper (the multiply is still FP16), but the bandwidth wins translate directly into decode speed.
FP4 and NF4 sit at the cutting edge. NF4 is a 4-bit float format designed for QLoRA and similar fine-tuning workflows. FP4 is a hardware-native format on B200 tensor cores, so both bandwidth and compute drop together. At this end of the ladder the quality cost starts to become visible without QAT, but for many serving targets it remains acceptable.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM and SGLang both ship FP8 weight quantization as a one-flag option for Llama 4 and Mistral Large 3, halving HBM use with near-zero accuracy loss.
- AWQ and GPTQ at INT4 are the standard ways to fit a 70B model onto a single 80 GB H100, used by every major open-weight serving stack.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does weight quantization help decode more than prefill?
Decode is bandwidth-bound because it streams the entire weight stack per token. Prefill is compute-bound, processing many tokens in parallel against the same weights with much higher arithmetic intensity, so shrinking bytes per weight buys less.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking quantization shrinks everything in the model. Only the stored weights move to a lower bit-width; activations typically stay in FP16 or BF16, and the matmul may even upcast back to higher precision before doing the multiply.
60 second bullets to scan on the way to the call.
Identify which tensor changes format and which usually does not.
Recite the bytes per weight ladder from FP16 down to 4-bit formats.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.