W4A16 plus an FP8 KV cache cuts both large per-step HBM reads (weights and KV) while keeping activations in FP16 to protect logits. Quality loss stays under 1%, and decode bytes drop by up to about 4x.
Imagine a cook who must fetch two heavy crates from a faraway pantry for every plate. One crate holds ingredients (the model's learned numbers), and one holds notes (the history of the conversation so far). The best improvement is to shrink both crates without changing the taste. The best recipe packs the ingredients into a quarter-sized crate with careful compression that keeps the taste (the W4A16 method). It also packs the notes crate to half size (an FP8 KV cache). The other choices are worse. Squeezing everything to 4 bits, including the seasoning added while cooking, makes the food taste wrong. Squeezing only the salt shaker leaves both crates huge. Halving the ingredients crate while also compressing the seasoning gives a smaller gain and some loss of flavor.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Quantization choice on a decode-bound workload is a case where the instinct 'smaller numbers everywhere is better' causes a real quality regression. The right choice depends on which bytes dominate the per-step HBM read and which tensors tolerate precision loss best. The two are uneven: weights tolerate INT4 well, activations lose quality much faster, and the KV cache sits between them.
W4A16 weight-only quantization plus an FP8 KV cache is a common production setup built on this difference. It is not the most aggressive recipe possible. It gives the most bandwidth reduction per unit of quality cost. More aggressive choices (INT4 activations, 4-bit KV) compound precision loss through the layers and hurt long-context reasoning. Less aggressive choices (W8A8, FP8 weights) leave bandwidth gains unused.
The gain also depends on kernels: a fused dequantize-matmul kernel is what turns the smaller weights into faster decode in a serving stack.
The per-step HBM byte budget
Pick a 70B-class model in FP16. The weight matrices sum to about 140 GB. The KV cache, in FP16, stores 2 vectors per layer per KV head per token of context. On a model with 80 layers and grouped-query attention (8 KV heads, head_dim 128), the per-token KV footprint is 80 * 8 * 128 * 2 * 2 bytes = 320 KB. A 64K-token context is therefore about 20 GB of KV cache for that one request.
At 3.35 TB/s (one H100's HBM bandwidth), streaming 140 GB takes about 42 ms. Streaming 20 GB of KV takes another 6 ms. These reads dominate each decode step. Compute for one token is sub-millisecond. About 160 GB per step in the long-context case is what bandwidth-bound decode means.
Quantization reduces this number directly. W4 cuts the weight term to about 35 GB. FP8 KV cuts the KV term to about 10 GB. The new total is about 45 GB per step, taking about 13 ms, a 3.5x decode speedup on a long-context workload. Combined with continuous batching, this is a common production setup.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM serves AWQ and GPTQ INT4 checkpoints with Marlin kernels (awq_marlin, gptq_marlin) on Ampere and Hopper GPUs.
- NVIDIA TensorRT LLM lists W4A16 AWQ and GPTQ recipes and an FP8 KV cache recipe in its quantization docs.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the kernel matter as much as the quantization method for W4A16?
W4 weights must be dequantized to the activation precision before the multiply-accumulate. Naive implementations write the dequantized weights to memory and read them again. Fused kernels (Marlin in vLLM, TensorRT LLM's W4A16 kernels) dequantize inside the matmul and approach peak HBM bandwidth. A poorly tuned kernel can lose 20 to 30% of the bandwidth gain.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Quantizing activations aggressively to get a bigger bandwidth cut. Activation quantization (INT4, or INT8 in some layers) is where quality degrades fastest, because small precision losses in activations change the logits.
60 second bullets to scan on the way to the call.
Why decode is memory-bandwidth bound and what bytes per step looks like
Why weight-only quantization (W4A16) preserves quality better than weight+activation
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.