Match each weight/activation quant regime to its mechanism and best-fit workload

**Quantization** stores numbers in fewer bits. A weight that was 16 bits becomes 8 or 4 bits, so the model takes less memory and can move faster. The naming convention WxAy tells you the bits on weights (W) and activations (A). W8A8 means 8-bit weights AND 8-bit activations. Because both sides are integers, the actual matrix multiply can run on INT8 tensor cores, which are roughly twice as fast as FP16 cores. This helps when raw math throughput is the bottleneck. W4A16 means 4-bit weights but 16-bit activations. The weights are tiny in memory, but they get unpacked back to FP16 before the multiply. So the math is still FP16; you only saved on the bytes you had to read for the weights. Why two schemes? LLM serving has two phases with different limits. Prefill, reading the prompt, is compute-bound, so faster INT8 math helps. Decode, writing tokens one by one, is memory-bound, so a smaller weight read helps. W8A8 fits prefill; W4A16 fits decode.

**Quantization** choices follow the bottleneck of the phase. The WxAy notation gives weight bits then activation bits. The matching trick is to ask whether you are compute-bound or bandwidth-bound. W8A8 quantizes both sides to INT8, so the whole matmul runs on INT8 tensor cores. That doubles math throughput versus FP16. It shines in prefill and large-batch serving, where you are compute-bound. The canonical method is SmoothQuant. It migrates activation outliers into the weights so INT8 activations stay accurate. W4A16 quantizes only the weights to 4-bit and keeps activations in FP16. The weights are dequantized to FP16 before the multiply, so math runs in FP16. The only win is a 4x smaller weight read from HBM. That is what decode at small batch needs, because decode is memory-bandwidth-bound. The canonical methods are AWQ and GPTQ, both weight-only with group-wise scales. So decode at batch 1 on a 70B model picks W4A16. Bandwidth dominates there. Prefill or high-batch throughput picks W8A8, because at large batch the workload turns compute-bound and INT8 math wins.