Weight quantization in plain terms: what changes and what stays the same?

Attempt it

Weight quantization stores model parameters in a lower bit-width than the FP16 baseline (typically FP8, INT8, or INT4) instead of the higher-precision floats they were trained in. Activations stay in higher precision (often FP16 or BF16) and the matmul either runs in FP16 with dequantize on the fly, or in a low-bit tensor-core path if the hardware supports it. The win is a smaller HBM footprint and fewer bytes streamed per decode step, which is what speeds bandwidth-bound generation.

Your answer

Quick reference

TL;DR

Weight quantization stores model weights in a lower bit-width (FP8, INT8, INT4) than the FP16 baseline so each decode step streams fewer bytes from HBM; activations usually stay in higher precision and the matmul

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a giant library where every book is written in a large, fancy font that takes up lots of shelf space. Picture the same library rewritten in a smaller font. The story is the same, the words mean the same thing, but each book is thinner so you can grab one off the shelf faster. The reader (the GPU) still understands the smaller font fine. The lights and chairs in the library (the running-around numbers and the actual reading) stay normal-sized. The whole point is that fetching books from the shelf was the slowest step, so making the books thinner makes the whole library run faster. The technique that compresses a model's learned numbers this way is what people mean by weight compression.

Key concepts

Quantization Weight quantization FP8 INT4

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

2 min: weights shrink to FP8 / INT8 / INT4, activations stay FP16 / BF16, matmul either dequantizes on the fly or runs native low-bit on H100 / B200, and the win is fewer bytes streamed per decode step in a bandwidth-bound regime.

Real-world examples

Real products, models, and research that use this idea.

vLLM and SGLang both ship FP8 weight quantization as a one-flag option for Llama 4 and Mistral Large 3, halving HBM use with near-zero accuracy loss.
AWQ and GPTQ at INT4 are the standard ways to fit a 70B model onto a single 80 GB H100, used by every major open-weight serving stack.
NVIDIA's TensorRT-LLM uses native FP8 tensor cores on H100 and FP4 on B200, so weight quantization there saves both memory bandwidth and compute.
DeepSeek V4 ships FP8-quantized weights as a first-class deployment artifact, not a post-hoc conversion, treating FP8 as the production storage format.
QLoRA and similar fine-tuning workflows use NF4 (a 4-bit float format) to fit a frozen base model into much less VRAM during training.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does weight quantization help decode more than prefill?

Sign in or go Pro to see the approach for this follow-up.

QHow does W4A16 differ from W8A8, and when would you pick each?

Sign in or go Pro to see the approach for this follow-up.

QWhat is the difference between weight quantization and KV-cache quantization?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Thinking quantization shrinks everything in the model. Only the stored weights move to a lower bit-width; activations typically stay in FP16 or BF16, and the matmul may even upcast back to higher precision before doing the multiply.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.