Bandwidth Bound

Which quantization combo squeezes the most decode bandwidth per percent of quality lost?

Walk through the byte accounting that proves a single-batch decode step is bandwidth-bound on H100.

Predict the bandwidth-vs-compute latency of a single Llama-70B decode step on H100