How much does FP8 KV cache help, what does it cost and what is KIVI?

During decode, the model rereads the **KV cache** from GPU memory at every single step. Long context means a huge cache, and reading it is what makes each new token slow. So the cache costs you both memory and read bandwidth. Storing the cache in FP8 instead of FP16 cuts each stored number from two bytes to one. That halves the memory the cache needs and halves how much data the GPU streams per step. The freed memory lets you serve roughly twice as many requests at once, which is the real throughput win. The cost is small. You run a short calibration pass to pick scaling factors that map the floating-point values into the smaller format without clipping. After that, a fused kernel dequantizes and attends in one shot, so runtime overhead is minor. KIVI goes further, down to 2 bits. It uses a different scaling style for keys versus values because their numbers are shaped differently. The danger is quality: very aggressive quantization can blur similar past tokens, which hurts tasks like long-document retrieval or code where the exact stored value matters.

**FP8 KV cache** stores keys and values in one byte instead of two. That halves cache memory and halves the bandwidth the GPU reads per decode step. The bandwidth point is the real win. Decode is memory-bound, and every step streams the whole cache from HBM. Halving the bytes roughly halves the cache-read portion of per-step latency. The freed memory then converts directly into batch size. You fit about twice the requests at the same budget, which is the dominant throughput lever. The calibration cost is modest. You sweep activations to pick scales, either per-channel for keys or per-token for values, so the quantized range avoids clipping. Runtime overhead is one fused dequantize then attend kernel. The design then leans on asymmetry. KIVI pushes to 2 bits using per-channel scales for K and per-token scales for V. That is an extra 4x reduction beyond FP8, with under one point of quality drop in published results. Regressions appear on long-context retrieval, code, and exact-value tasks. Aggressive quant erodes the model's ability to discriminate similar past tokens, so precise recall degrades first.