Define activation quantization and explain how W8A8 differs from weight only quant.

A neural network has two kinds of tensors in memory at inference time. **Weights** are the static parameters learned during training. **Activations** are the intermediate values that flow between layers as a forward pass processes a token: the output of one layer's matmul becomes the input to the next. **Weight quantization** stores the weights in a low-precision format like INT8 or FP8 instead of BF16. This cuts the memory used to hold the model, but the matmul itself still runs in higher precision because the activations are still BF16. The runtime dequantizes the weights on the fly so they can multiply against the BF16 activations. **Activation quantization** stores the activations in low precision too. When you do both, the recipe is called **W8A8**: 8-bit weights and 8-bit activations. Now the matmul receives two 8-bit inputs, which means it can run on the GPU's INT8 or FP8 tensor cores directly. Compute roughly halves on top of the memory cut you already got from weight quantization. The trade-off is that activations have a wider dynamic range than weights, so quantizing them is harder. A few outlier channels can dominate the range and squash the rest. Techniques like SmoothQuant and per-channel scaling exist specifically to handle these outliers so W8A8 hits quality close to BF16.

**Activation quantization** stores the per-token tensors that flow between transformer layers (matmul inputs, residual streams, attention scores before softmax) in a low-precision format such as INT8 or FP8 instead of the BF16 they would normally use. Weight quantization is paired with it: the static parameters are also stored low-precision. The recipe **W8A8** means 8-bit weights and 8-bit activations. The reason it matters is what happens at matmul time. Modern GPUs (Hopper H100, Blackwell B200) have tensor cores that run INT8 and FP8 matmuls at substantially higher throughput than BF16. Weight-only quantization stores the weights small but still runs the matmul in BF16, because one operand is BF16; the GPU has to dequantize on the fly. W8A8 keeps both operands low-precision, so the matmul lands on the INT8 or FP8 tensor-core path natively. That roughly halves compute on top of the memory savings. The catch is that activations are dynamic. They depend on the input and have a wider, less predictable distribution than weights. A few outlier channels can dominate the per-tensor range and squash the rest, which is why naive W8A8 often regresses quality compared to weight-only. Production recipes handle this with per-channel or per-token scaling, outlier-aware techniques like **SmoothQuant** that migrate outlier magnitude from activations into weights, and quantization-aware calibration on a small held-out corpus. Done right, W8A8 hits quality within a fraction of a percent of BF16 while delivering near-2x compute throughput and lower memory bandwidth on every layer.