Which quantization combo squeezes the most decode bandwidth per percent of quality lost?

MCQ·Medium·4.0 · 0·~1 min·Asked atDescript NVIDIA

Attempt it

**Decode is bandwidth-bound.** Every step the GPU rereads the full weight matrix and the per-request KV cache from HBM and does very little compute per byte. Cutting the bytes per step is the dominant lever. Weights are the biggest term. **W4A16** quantization (AWQ, GPTQ) cuts weight bytes 4x by storing them in INT4 while keeping activations in FP16. The activations stay at full precision, so the matrix multiplications produce nearly the same logits as the original model. Sub-1% quality regression on most chat benchmarks is typical. The second-biggest term is the **KV cache**, which grows with sequence length. Quantizing it to FP8 halves its size and bandwidth without measurable quality loss in most workloads. The other options are worse. **W8A8** halves both terms (smaller win) and pays for activation quantization (some quality cost). **W4 with INT4 activations** is too aggressive, INT4 activations break long-context reasoning. **INT4 LM-head only** quantizes one small tensor and barely moves the bandwidth needle. The winning combo: W4A16 + FP8 KV. Roughly 5-6x bandwidth reduction on decode with sub-1% quality loss on standard evals.

**Decode-phase bandwidth budget** is dominated by two terms: streaming the full weight set from HBM each step, and reading the per-request KV cache. The right quantization recipe attacks both while preserving the precision of activations and logits. **W4A16** (weight-only INT4) is the central move. AWQ and GPTQ are calibration-based: they minimize the per-channel quantization error using a small calibration set, so the resulting model preserves chat-benchmark quality to within ~1%. The activations stay in FP16, so every matmul produces nearly-original logits. Per-step weight bytes drop ~4x, which is most of the bandwidth win. Stack on **FP8 KV cache** to halve the per-step KV read. KV bytes grow linearly with sequence length, so for long-context workloads the KV term can exceed the weight term, FP8 KV becomes proportionately more important as context grows. Quality regression is negligible on standard tasks. The wrong choices each have a specific failure mode. **W8A8** halves both terms (so weaker bandwidth win) and pays an activation-quantization cost, which empirically degrades quality on reasoning and long-context tasks more than weight-only quant. **INT4 activations** routinely break long-context reasoning, as the precision loss compounds across attention layers. **LM-head-only INT4** quantizes one tiny tensor and barely changes the byte budget. Production recipe in 2026: W4A16 (AWQ or GPTQ) plus FP8 KV on H100/B200 hardware, with vLLM or TensorRT-LLM as the serving stack.

TL;DR

W4A16 plus FP8 KV cache attacks both dominant per-step HBM byte terms (weights and KV) while leaving activations in FP16 to protect logit accuracy. Sub-1% quality loss on chat benchmarks, ~5-6x bandwidth win on decode.

Memory aid

Easy to grasp

Imagine the cook has to fetch two heavy crates from a faraway pantry on every plate: a crate of ingredients (the model's learned numbers) and a crate of running notes (the in-flight history of the conversation so far). The cheapest improvement is shrinking both crates without making the food taste different. The best recipe: pack the ingredients into a quarter-sized crate using a clever compression that preserves taste (the W4A16 trick), and squash the notes crate to half size (FP8 history). Now picture the bad alternatives. Squeeze everything down to 4-bit including the in-flight seasoning, the food tastes wrong. Squeeze only the salt shaker tighter, the crates are still huge. Squeeze both crates to half size but the new ingredient compression also degrades flavor, partial win, real cost.

Key concepts

Quantization KV cacheMemory bandwidth boundAWQ

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Quantization choice on a decode-bound workload is one of the questions where the wrong instinct, 'smaller numbers everywhere is better', produces a real quality regression that takes weeks to debug. The right framing requires understanding which bytes dominate the per-step HBM read and which tensors tolerate precision loss best. The answer turns out to be highly asymmetric: weights tolerate aggressive INT4 quantization remarkably well, while activations break much faster, and the KV cache sits between them.

The production recipe in 2026, W4A16 weight-only quantization plus FP8 KV cache, emerged from years of iteration on this asymmetry. It is not the most aggressive quantization recipe possible; it is the one that maximizes bandwidth reduction per unit of quality cost. Going more aggressive (INT4 activations, INT4 KV) starts to compound precision losses through attention layers and degrades long-context reasoning. Going less aggressive (W8A8, FP8 weights) leaves bandwidth wins on the table.

This deep dive lays out the byte budget per decode step, walks the quality curves for weight vs activation vs KV quantization, explains why AWQ and GPTQ outperform naive rounding, and connects the recipe to the kernel-level work that makes the theoretical bandwidth win actually realizable in production serving stacks.

The per-step HBM byte budget

Pick a 70B-class model in FP16. The weight matrices sum to about 140 GB. The KV cache, in FP16, stores 2 vectors per layer per head per token of context. On a model with 80 layers and group-query attention (8 KV heads), the per-token KV footprint is roughly 80 * 8 * 128 * 2 * 2 = 320 KB. A 64K-token context per request is therefore about 20 GB of KV cache for that one request.

On an H100 with 3.35 TB/s of HBM bandwidth, streaming 140 GB takes about 42ms. Streaming 20 GB of KV takes another 6ms. Each decode step is dominated by these reads; compute for one token is sub-millisecond. The bytes per step number, 160-180 GB for the long-context case, is what bandwidth-bound decode actually means.

Quantization is a direct attack on this number. W4 cuts the weight term to ~35 GB. FP8 KV cuts the KV term to ~10 GB. The new bytes per step is ~45 GB, taking about 13ms, a 3.5x decode acceleration on a long-context workload. Stack this with continuous batching and you have the standard 2026 production recipe.

Why W4A16 beats W8A8 on quality

The KV cache term and FP8

Kernels turn theory into delivered bandwidth

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

vLLM's quantization page lists W4A16 (AWQ, GPTQ) plus FP8 KV as the recommended H100 recipe for Llama and Qwen.
NVIDIA TensorRT-LLM ships Marlin and FP8 kernels specifically to keep W4A16 dequantize cost off the critical path.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the kernel matter as much as the quantization method for W4A16?

W4 weights must be dequantized back to the activation precision before the FMAC. Naive implementations waste shared memory and re-read weights. Specialized kernels (Marlin, FlashLLM, TRT-LLM custom kernels) keep dequantize-fused matmul on the fast path and approach peak HBM bandwidth. A bad kernel can give back 20-30% of the bandwidth win.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Quantizing activations aggressively to chase a bigger bandwidth number. Activation quantization (INT4 or even INT8 in some layers) is where quality degrades fastest because logits become sensitive to small precision losses.

60 second bullets to scan on the way to the call.

Why decode is memory-bandwidth bound and what bytes per step looks like
Why weight-only quantization (W4A16) preserves quality better than weight+activation

Primary sources. Browse if you want the original framing.