An engineer says 'FP8 KV cache just halves memory, no other benefit.' Which response is most correct?

The right answer is option B. The engineer is correct that FP8 KV cache halves the memory footprint, but that is only half the picture. Decode is the slow, one token at a time phase of generation. At every step the model reads the entire KV cache out of HBM (the GPU's high-bandwidth memory). This read is the bottleneck. Decode is **memory-bandwidth bound**, meaning the GPU spends more time moving bytes than doing math. Now shrink each cached element from two bytes (bf16) to one byte (fp8). You did not just halve how much space the cache takes. You also halved how many bytes have to be read every single step. Fewer bytes to move means each decode step finishes faster, so latency improves. There is a third effect. Because each request's cache now fits in half the space, the scheduler can pack roughly twice as many requests onto the same GPU. That raises throughput on top of the per-request latency win. So the change touches three things at once: memory, latency, and batch size. Calling it 'just memory' misses the two effects that matter most for serving cost.

Option B is correct. The engineer captured the static memory saving but missed the bandwidth saving, which is the more interesting half. Decode is memory-bandwidth bound. Each step reads the full KV history from HBM to compute attention. The KV cache is the dominant byte source at long context. Halving its dtype from bf16 to fp8 halves those bytes read per step. On the bandwidth slope of the roofline, fewer bytes means a faster step, so decode latency drops directly. The footprint halving has a second consequence. Freed HBM lets the scheduler admit roughly twice as many concurrent requests. That raises aggregate throughput, which compounds with the per-step latency win. The distractors are instructive. Option C inverts the cost. Dequant is cheap and fused into the attention kernel, so it does not add net bandwidth. Option D is simply false. FP8 KV is a post-training quantization scheme with per-channel or per-token scale calibration. It needs no retraining. The one real cost is accuracy. FP8 KV is usually near-lossless. Aggressive INT4 KV can degrade quality at very long context. That is the tradeoff to flag, not bandwidth.