What does W8A8 vs W4A16 mean and which regime is appropriate for memory-bound decode?

Attempt it

Define W8A8 and W4A16 quantization regimes. Explain which one is appropriate for memory-bandwidth-bound LLM decode and why, and which fits compute-bound prefill at large batch.

Free · 2 AI evals / day

Quick reference

TL;DR

W8A8 runs INT8 weights and activations on INT8 tensor cores for compute-bound prefill throughput; W4A16 shrinks the weight read 4x for memory-bound decode latency.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a chef cooking from a giant recipe book. Two things slow them down: running to a far-off shelf to fetch recipe pages, and the actual chopping and stirring. When the kitchen is quiet and one dish is cooking, the slow part is the trip to the shelf for each page. Make the book tiny and those trips get fast. That is W4A16: shrink the recipe (the weights) so fetching is cheap. When the kitchen is slammed with a hundred dishes at once, the shelf trips get shared across dishes, and now the chopping is the bottleneck. So you hire faster hands instead. That is W8A8: keep both ingredients and recipe lean so a faster cooking station can run flat out. The whole trick is matching the fix to whichever cost actually dominates.

Key concepts

Quantization W8A8 INT8 quantization W4A16 weight-only quantization Memory-bound decode

Quick scan · 2–4 min

Everything important, quickly.

4 min: decode the WxAy notation, classify decode versus prefill on the roofline, map W4A16 to bandwidth and W8A8 to compute, name SmoothQuant and AWQ, then close on routing by batch size.

Side-by-side

Property	W8A8	W4A16
Weight precision	INT8 (1 byte)	INT4 (0.5 byte)
Activation precision	INT8	FP16
Matmul execution	INT8 tensor cores	FP16 after dequant
Compute throughput	~2x FP16	Same as FP16
Weight bytes read	2x smaller	4x smaller
Canonical method	SmoothQuant	AWQ or GPTQ
Best-fit phase	Compute-bound prefill, large batch	Bandwidth-bound decode

Real-world examples

Real products, models, and research that use this idea.

vLLM serves AWQ and GPTQ W4A16 checkpoints of Llama 4 and Qwen 3 to cut per-token decode latency at low batch sizes.
NVIDIA TensorRT-LLM exposes W8A8 INT8 and FP8 paths that light up Hopper and Blackwell tensor cores for compute-bound prefill.
SmoothQuant is the canonical recipe production stacks use to make W8A8 viable by migrating activation outliers into weights.
DeepSeek V4 and Mistral Large 3 ship quantized weight-only variants so commodity GPUs can run decode within HBM bandwidth limits.
The AutoAWQ and GPTQModel toolchains on Hugging Face generate the group-scaled 4-bit weights that W4A16 serving consumes.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does W4A16 give no speedup on a compute-bound prefill even though weights are 4x smaller?

Sign in or go Pro to see the approach for this follow-up.

QWhy do activations need SmoothQuant for W8A8 but weights tolerate INT4 with simpler methods?

Sign in or go Pro to see the approach for this follow-up.

QHow would you serve one model to both interactive low-batch users and bulk prefill jobs?

Sign in or go Pro to see the approach for this follow-up.

QWhere does FP8 fit relative to W8A8 and W4A16 on a 2026 Blackwell GPU?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.