What does W8A8 vs W4A16 mean and which regime is appropriate for memory-bound decode?

Quantization stores numbers in fewer bits to make a model smaller and faster. The naming convention WxAy tells you two things: how many bits the **weights** use (the W) and how many bits the activations use (the A). So W8A8 means 8-bit weights and 8-bit activations, while W4A16 means 4-bit weights with 16-bit activations. W8A8 puts both sides into INT8. That lets the matmul run on the chip's INT8 tensor cores, which do roughly twice the math per second as the FP16 path. The catch is that activations have nasty outlier values, so a method called SmoothQuant shifts that difficulty into the weights to keep accuracy. W4A16 squeezes only the weights down to 4 bits and leaves activations at 16 bits. At runtime the 4-bit weights are expanded back to FP16 just before the matmul, so the math is still FP16. Methods like AWQ and GPTQ produce these 4-bit weights with per-group scale factors. Which one you pick depends on the bottleneck. When generating one token at a time, the slow part is reading weights from memory, so fewer weight bytes wins. For a big prompt or large batch, raw math is the slow part, so faster INT8 cores win.

**W8A8** means 8-bit weights and 8-bit activations. Both operands enter the matmul as INT8. That activates the GPU's INT8 tensor cores, which deliver roughly double the FP16 math throughput. SmoothQuant migrates activation outliers into the weights so both sides quantize cleanly. W4A16 means 4-bit weights and 16-bit activations. The kernel dequantizes weights to FP16 on the fly, then runs FP16 math. AWQ and GPTQ are the standard production paths, emitting 4-bit weights with group-wise scales. The choice follows the bottleneck of the phase. Decode generates one token per step. Its cost is dominated by streaming weights from HBM. The weight bytes read per step are: $$\text{bytes/step} \approx N_{\text{params}} \cdot b_w$$ Going from FP16 weights at 2 bytes to INT4 at half a byte shrinks that read 4x. Lower bit-width directly cuts per-step decode latency. Activations stay FP16 because the workload is bandwidth-bound, not compute-bound, so there is no throughput pressure. Prefill and large-batch serving are compute-bound. Weight reads amortize across many tokens, so raw matmul throughput dominates. There W8A8 wins, because INT8 tensor cores double the math rate end to end. Rule of thumb: pick weight bytes for decode, pick INT8 compute for prefill.