Define activation quantization and explain how W8A8 differs from weight-only quant.

Attempt it

Your answer

A neural network has two kinds of tensors in memory at inference time. **Weights** are the static parameters learned during training. **Activations** are the intermediate values that flow between layers as a forward pass processes a token: the output of one layer's matmul becomes the input to the next. **Weight quantization** stores the weights in a low-precision format like INT8 or FP8 instead of BF16. This cuts the memory used to hold the model, but the matmul itself still runs in higher precision because the activations are still BF16. The runtime dequantizes the weights on the fly so they can multiply against the BF16 activations. **Activation quantization** stores the activations in low precision too. When you do both, the recipe is called **W8A8**: 8-bit weights and 8-bit activations. Now the matmul receives two 8-bit inputs, which means it can run on the GPU's INT8 or FP8 tensor cores directly. Compute roughly halves on top of the memory cut you already got from weight quantization. The trade-off is that activations have a wider dynamic range than weights, so quantizing them is harder. A few outlier channels can dominate the range and squash the rest. Techniques like SmoothQuant and per-channel scaling exist specifically to handle these outliers so W8A8 hits quality close to BF16.

**Activation quantization** stores the per-token tensors that flow between transformer layers (matmul inputs, residual streams, attention scores before softmax) in a low-precision format such as INT8 or FP8 instead of the BF16 they would normally use. Weight quantization is paired with it: the static parameters are also stored low-precision. The recipe **W8A8** means 8-bit weights and 8-bit activations. The reason it matters is what happens at matmul time. Modern GPUs (Hopper H100, Blackwell B200) have tensor cores that run INT8 and FP8 matmuls at substantially higher throughput than BF16. Weight-only quantization stores the weights small but still runs the matmul in BF16, because one operand is BF16; the GPU has to dequantize on the fly. W8A8 keeps both operands low-precision, so the matmul lands on the INT8 or FP8 tensor-core path natively. That roughly halves compute on top of the memory savings. The catch is that activations are dynamic. They depend on the input and have a wider, less predictable distribution than weights. A few outlier channels can dominate the per-tensor range and squash the rest, which is why naive W8A8 often regresses quality compared to weight-only. Production recipes handle this with per-channel or per-token scaling, outlier-aware techniques like **SmoothQuant** that migrate outlier magnitude from activations into weights, and quantization-aware calibration on a small held-out corpus. Done right, W8A8 hits quality within a fraction of a percent of BF16 while delivering near-2x compute throughput and lower memory bandwidth on every layer.

Quick reference

TL;DR

Activation quantization stores the per-token tensors flowing between layers in INT8 or FP8. W8A8 quantizes both weights and activations, so the matmul itself runs at low precision, halving compute as well as memory.

Memory aid

Easy to grasp

Imagine a factory line. The machines are the model's learned coefficients, and the half-finished products moving from station to station are the in-flight numbers between layers. Compressing only the machines into smaller cabinets saves floor space, but the products are still passed around in big crates that the machines have to unpack before each step. W8A8 is the recipe that also compresses the crates into compact 8-bit packages, so each station receives small parcels and works on them directly. Less storage, faster handoff, less unpacking. The trade-off is that the smaller crates have less precision; rare odd-shaped products can get squashed in transit, which is why the compression step needs careful calibration to keep flavor.

Key concepts

Quantization Activation quantization W8A8 quantization

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

2 min: weights versus activations + why weight-only stays at BF16 matmul + how W8A8 unlocks low-precision tensor cores + outlier handling + INT8 versus FP8 in 2026.

Real-world examples

Real products, models, and research that use this idea.

NVIDIA TensorRT-LLM ships FP8 W8A8 as the default low-precision recipe on H100 and B200 for Llama 4, Qwen 3, and DeepSeek V4 deployments.
vLLM supports both INT8 W8A8 (with SmoothQuant calibration) and FP8 W8A8 as first-class quantization paths for production serving.
SmoothQuant (Xiao et al.) is the canonical outlier-handling technique that made W8A8 viable on transformer activations.
GPTQ and AWQ are weight-only recipes that pair well with W8A8 activation quantization for deployment stacks that need both bandwidth and compute wins.
Anthropic, OpenAI, and DeepSeek production stacks all ship FP8 W8A8 internally to extract the maximum throughput from their Hopper and Blackwell fleets.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does SmoothQuant migrate outlier magnitude from activations to weights instead of clipping them?

QHow does W8A8 interact with FP8 KV cache?

QWhen would you choose mixed precision (some layers BF16, most W8A8) over uniform W8A8?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Believing weight-only quantization gives the same speedup as W8A8. Weight-only cuts memory, but the matmul still runs in FP16. Only W8A8 reaches the INT8 or FP8 tensor-core path.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.