How much does FP8 KV cache help, what does it cost and what is KIVI?
Quantizing the KV cache is the second big lever (after weight quantization) for serving long context. Quantify what FP8 KV cache buys you, explain the calibration cost, and describe how KIVI pushes the trade further. When does KV-cache quant cause noticeable quality regressions?
FP8 KV cache halves cache bytes and bandwidth, roughly doubling batch size. KIVI pushes to 2 bits via per-channel K and per-token V scales, with quality risk on long-context retrieval.
Imagine a giant warehouse of notes the model rereads to write each new word. The notes are stored at high precision, which eats space and slows every read. Quantizing the cache is like writing the notes in shorthand: each note takes half the room, so you can hold twice as many conversations at once, and reading them is faster too. The catch is that shorthand loses detail. For casual notes that is fine. But if a note records an exact number, a variable name, or a precise fact buried deep in a long document, the shorthand might blur it. The model then struggles to tell two similar past notes apart, and the answer drifts on tasks where exact recall matters.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
KV-cache quantization is the second major precision lever in LLM serving, after weight quantization. Once weights are squeezed to FP8 or INT4, the KV cache becomes the dominant consumer of memory and bandwidth at long context. A 70B model at 128k context can spend more HBM on the cache than on its own weights, and during decode the cache is reread in full at every step.
That reread is the crux. Decode is memory-bandwidth bound, not compute-bound, so the bytes you stream per step set your latency floor. The arithmetic units on a modern GPU sit mostly idle during decode, waiting on HBM. Shrinking the stored precision of K and V attacks both axes at once: it frees memory for larger batches and it cuts the per-step bandwidth read. The question a senior interviewer probes is whether you understand that these are two distinct wins, how the calibration is done, how far schemes like KIVI push the trade, and exactly where the quality cliffs are.
Weight quantization and KV-cache quantization are often conflated, but they target different costs. Weight quant shrinks a fixed footprint that is shared across every request in a batch. KV quant shrinks a per-request footprint that grows with sequence length, so its payoff scales with context and concurrency. At short context the weights dominate; at long context the cache dominates, and that is exactly the regime where serving economics get painful.
This deep dive quantifies FP8 KV savings, walks through per-channel versus per-token scaling, explains KIVI's asymmetric 2-bit design, and maps the failure modes so you can reason about when the throughput is worth the accuracy risk.
What FP8 KV cache actually buys
The KV cache size follows a fixed formula. Storing it in FP8 instead of FP16 changes only the bytes per element term:
With b dropping from 2 to 1, the cache halves. That is the obvious memory win. The less obvious and more important win is bandwidth. During decode, the attention layer reads the entire cache from HBM at every step, so the bytes streamed per token also halve. Because decode latency is dominated by these reads, the cache-read portion of per-step time falls by roughly half.
The freed memory does not just sit idle. In a serving system you immediately spend it on a larger batch. Doubling the resident requests roughly doubles throughput, because batching amortizes the fixed weight reads across more sequences. This is why FP8 KV is described as a throughput lever, not merely a memory optimization. Memory, bandwidth, and batch size are three faces of the same saving.
FP8 specifically is attractive because the format keeps an exponent field. The two common layouts, E4M3 and E5M2, trade mantissa bits for dynamic range. With an exponent the format spans several orders of magnitude, so the occasional large activation does not blow out the whole tensor. That is why FP8 KV usually drops well under a point of quality with almost no tuning, while pushing the same bytes into a 1-byte integer needs careful per-channel scales to match it.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM ships FP8 KV cache as a flag, letting operators trade a small quality check for roughly double the concurrent requests on long-context workloads.
- TensorRT-LLM (NVIDIA) exposes FP8 and INT8 KV cache with calibration on H100 and B200, pairing it with paged attention for production decode.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy do keys prefer per-channel scaling while values prefer per-token scaling?
Look at the outlier structure. Key activations show persistent large-magnitude channels, so a scale per channel bounds error well. Value activations vary more across tokens than channels, so a scale per token fits their spread. Matching granularity to where the variance lives minimizes quantization error.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming KV quant saves only memory. It also halves the bandwidth read per decode step, which is the actual bottleneck, and the freed memory converts directly into bigger batches and higher throughput.
60 second bullets to scan on the way to the call.
Why KV quant saves bandwidth as well as memory during decode
How freed cache memory translates into larger batch size
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.