Which quantization combo squeezes the most decode bandwidth per percent of quality lost?

MCQ·Medium·4.0 · 0·~1 min·Asked atNVIDIA

Attempt it

**Decode is bandwidth-bound.** Every step the GPU rereads the full weight matrix and the per-request KV cache from HBM and does very little compute per byte. Cutting the bytes per step is the dominant lever. Weights are the biggest term. **W4A16** quantization (AWQ, GPTQ) cuts weight bytes 4x by storing them in INT4 while keeping activations in FP16. The activations stay at full precision, so the matrix multiplications produce nearly the same logits as the original model. Sub-1% quality regression on most chat benchmarks is typical. The second-biggest term is the **KV cache**, which grows with sequence length. Quantizing it to FP8 halves its size and bandwidth without measurable quality loss in most workloads. The other options are worse. **W8A8** halves both terms (smaller win) and pays for activation quantization (some quality cost). **W4 with INT4 activations** is too aggressive, INT4 activations break long-context reasoning. **INT4 LM-head only** quantizes one small tensor and barely moves the bandwidth needle. The winning combo: W4A16 + FP8 KV. Roughly 5-6x bandwidth reduction on decode with sub-1% quality loss on standard evals.

**Decode-phase bandwidth budget** is dominated by two terms: streaming the full weight set from HBM each step, and reading the per-request KV cache. The right quantization recipe attacks both while preserving the precision of activations and logits. **W4A16** (weight-only INT4) is the central move. AWQ and GPTQ are calibration-based: they minimize the per-channel quantization error using a small calibration set, so the resulting model preserves chat-benchmark quality to within ~1%. The activations stay in FP16, so every matmul produces nearly-original logits. Per-step weight bytes drop ~4x, which is most of the bandwidth win. Stack on **FP8 KV cache** to halve the per-step KV read. KV bytes grow linearly with sequence length, so for long-context workloads the KV term can exceed the weight term, FP8 KV becomes proportionately more important as context grows. Quality regression is negligible on standard tasks. The wrong choices each have a specific failure mode. **W8A8** halves both terms (so weaker bandwidth win) and pays an activation-quantization cost, which empirically degrades quality on reasoning and long-context tasks more than weight-only quant. **INT4 activations** routinely break long-context reasoning, as the precision loss compounds across attention layers. **LM-head-only INT4** quantizes one tiny tensor and barely changes the byte budget. Production recipe in 2026: W4A16 (AWQ or GPTQ) plus FP8 KV on H100/B200 hardware, with vLLM or TensorRT-LLM as the serving stack.

Quick reference

TL;DR

W4A16 plus FP8 KV cache attacks both dominant per-step HBM byte terms (weights and KV) while leaving activations in FP16 to protect logit accuracy. Sub-1% quality loss on chat benchmarks, ~5-6x bandwidth win on decode.

Memory aid

Easy to grasp

Imagine the cook has to fetch two heavy crates from a faraway pantry on every plate: a crate of ingredients (the model's learned numbers) and a crate of running notes (the in-flight history of the conversation so far). The cheapest improvement is shrinking both crates without making the food taste different. The best recipe: pack the ingredients into a quarter-sized crate using a clever compression that preserves taste (the W4A16 trick), and squash the notes crate to half size (FP8 history). Now picture the bad alternatives. Squeeze everything down to 4-bit including the in-flight seasoning, the food tastes wrong. Squeeze only the salt shaker tighter, the crates are still huge. Squeeze both crates to half size but the new ingredient compression also degrades flavor, partial win, real cost.

Key concepts

Quantization KV cacheMemory bandwidth boundAWQ

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: decode bandwidth bottleneck + W4A16 as the highest-ROI quantization + FP8 KV as the secondary lever + why activation quantization is dangerous + the production recipe in 2026.

Real-world examples

Real products, models, and research that use this idea.

vLLM's quantization page lists W4A16 (AWQ, GPTQ) plus FP8 KV as the recommended H100 recipe for Llama and Qwen.
NVIDIA TensorRT-LLM ships Marlin and FP8 kernels specifically to keep W4A16 dequantize cost off the critical path.
DeepSeek V3 / V4 release notes in 2026 emphasize FP8 native weights + FP8 KV cache as the deployment baseline.
Llama 3.1 70B in W4A16 + FP8 KV runs comfortably on a single H100 80GB, where FP16 + FP16 KV does not.
Mistral Large 2 deployments at Together AI and Fireworks use W4A16 quantization with sub-1% MMLU regression vs the FP16 reference.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the kernel matter as much as the quantization method for W4A16?

QAt what context length does the FP8 KV term overtake the weight term in per-step bandwidth?

QHow does FP4 (introduced by Blackwell B200) change the recipe?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Quantizing activations aggressively to chase a bigger bandwidth number. Activation quantization (INT4 or even INT8 in some layers) is where quality degrades fastest because logits become sensitive to small precision losses.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.