Drag each answer to line up with its matching prompt
W8A8 (8-bit weights, 8-bit activations)
Minimizes bytes per token in the weight read with FP16 math; best for memory bandwidth bound decode
W4A16 (4-bit weights, 16-bit activations)
AWQ or GPTQ: weight only quantization with group wise scales
W8A8 typical method
Uses INT8 tensor cores end to end; best for compute bound prefill at large batch
W4A16 typical method
SmoothQuant: migrates activation outliers into weights so INT8 math stays accurate
Decode at batch 1 on a 70B model
Picks W4A16: bandwidth dominates, INT4 weights shrink HBM read 4×
W8A8 quantizes weights and activations to INT8 for compute-bound prefill on INT8 tensor cores; W4A16 shrinks only the weight read to 4-bit for bandwidth-bound decode.
Imagine a kitchen with two very different rushes. The lunch rush sends hundreds of identical orders at once, so the bottleneck is how fast the cooks can chop. You hire faster cooks who work in a coarser, quicker style. That is W8A8: everything runs in a faster, lower-precision mode because raw cooking speed is the limit. The late-night rush sends one order at a time, and the bottleneck is the long walk to the pantry for ingredients. You shrink the ingredients so each trip carries less weight. That is W4A16: you compress only what you fetch from storage, but cook it carefully in full precision. Same kitchen, two regimes, each matched to where the real wait is.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Quantization is the highest-leverage knob in LLM serving after the KV cache, and the W8A8 versus W4A16 distinction is where most candidates fall apart. The trap is treating quantization as a single make-it-smaller dial. It is not. The two regimes optimize two completely different resources, and the right choice is dictated entirely by which resource your workload is bottlenecked on. Get the resource wrong and your carefully calibrated low-bit model runs no faster than the FP16 baseline you started from.
The notation WxAy reads as x bits on weights, y bits on activations. W8A8 puts both at INT8. W4A16 puts weights at INT4 and leaves activations at FP16. That single difference in the activation precision decides whether the matmul can run on integer tensor cores or must fall back to FP16, and that in turn decides which phase of inference the scheme actually accelerates. Everything else in this question, the methods and the workloads, follows from that one fork.
This deep dive develops the roofline intuition first, then walks each regime through its mechanism, its enabling quantization method, and its best-fit workload. By the end you should be able to look at a serving scenario, classify it as compute-bound or bandwidth-bound, and name the quantization regime that wins, including the crossover where the answer flips as you scale batch size and as new hardware adds low-bit compute formats.
The roofline framing: compute-bound versus bandwidth-bound
Every GPU kernel sits somewhere on a roofline. Either it is limited by how fast the cores can do arithmetic, or by how fast it can move bytes to and from HBM. The deciding quantity is arithmetic intensity: FLOPs performed per byte read. Below the machine's ops-per-byte ratio you are memory-bound, above it you are compute-bound, and the binding resource is the only one worth optimizing.
LLM inference splits cleanly along this line. Prefill processes the whole prompt in parallel, so each weight is loaded once and reused across many token positions in the same forward pass. Arithmetic intensity is high, and the kernel is compute-bound. Decode generates one token at a time, so at batch 1 each weight is read from HBM and used for exactly one token's worth of math. Arithmetic intensity collapses toward one FLOP per byte, and the kernel is memory-bandwidth-bound. The tensor cores sit largely idle while the memory subsystem works flat out.
This is the whole basis for the matching question. A quantization regime that doubles math throughput only helps a compute-bound kernel. A regime that quarters the weight read only helps a bandwidth-bound kernel. Optimizing the non-binding axis is invisible at the wall clock. Mismatch the regime to the phase and you spend real calibration and kernel-engineering complexity for no speedup, which is precisely the mistake the question is built to expose.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | W8A8 | W4A16 |
|---|---|---|
| Weight bits | 8 (INT8) | 4 (INT4) |
| Activation bits | 8 (INT8) | 16 (FP16) |
| Matmul precision | INT8 tensor cores | FP16 (weights dequantized first) |
| Primary win | 2x math throughput | 4x smaller weight read |
| Bottleneck it relieves | Compute-bound | Memory-bandwidth-bound |
| Best-fit phase | Prefill, large-batch serving | Decode at small batch |
| Canonical method | SmoothQuant | AWQ, GPTQ |
Real products, models, and research that use this idea.
- vLLM and SGLang in 2026 ship AWQ and GPTQ W4A16 kernels as the default for low-batch decode of Llama 4 and Qwen 3.
- TensorRT-LLM (NVIDIA) exposes SmoothQuant W8A8 INT8 GEMMs tuned for compute-bound prefill on H100 and B200.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does W4A16 give zero matmul speedup despite using 4-bit weights?
Trace the kernel. Weights are dequantized back to FP16 in registers before the GEMM, so the actual arithmetic is FP16. The only saving is the HBM read of packed 4-bit weights. Compute throughput is unchanged.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating all quantization as one knob. The phase decides: prefill is compute-bound and wants INT8 math, decode is bandwidth-bound and wants the smallest weight read, regardless of compute precision.
60 second bullets to scan on the way to the call.
Why decode is bandwidth-bound and prefill is compute-bound
What WxAy notation encodes and what each axis controls
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.