How much does FP8 KV cache help, what does it cost and what is KIVI?

Attempt it

Quantizing the KV cache is the second big lever (after weight quantization) for serving long-context. Quantify what FP8 KV cache buys you, explain the calibration cost, and describe how KIVI pushes the trade further. When does KV-cache quant cause noticeable quality regressions?

Free · 2 AI evals / day

Quick reference

TL;DR

FP8 KV cache halves cache bytes and bandwidth, roughly doubling batch size. KIVI pushes to 2 bits via per-channel K and per-token V scales, with quality risk on long-context retrieval.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a giant warehouse of notes the model rereads to write each new word. The notes are stored at high precision, which eats space and slows every read. Quantizing the cache is like writing the notes in shorthand: each note takes half the room, so you can hold twice as many conversations at once, and reading them is faster too. The catch is that shorthand loses detail. For casual notes that is fine. But if a note records an exact number, a variable name, or a precise fact buried deep in a long document, the shorthand might blur it. The model then struggles to tell two similar past notes apart, and the answer drifts on tasks where exact recall matters.

Key concepts

KV cache quantization FP8 precision KIVI 2-bit KV cache Per-channel vs per-token scaling

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: FP8 savings (memory plus bandwidth), then batch size, then calibration scales, then the KIVI 2-bit asymmetric scheme and recent-token window, then failure modes on long-context retrieval and code.

Real-world examples

Real products, models, and research that use this idea.

vLLM ships FP8 KV cache as a flag, letting operators trade a small quality check for roughly double the concurrent requests on long-context workloads.
TensorRT-LLM (NVIDIA) exposes FP8 and INT8 KV cache with calibration on H100 and B200, pairing it with paged attention for production decode.
KIVI demonstrated 2-bit KV cache with per-channel K and per-token V scales, reporting near-lossless quality on most LLM benchmarks.
DeepSeek V4 combines Multi-head Latent Attention with low-precision KV storage so 128k-context serving stays within HBM budget.
SGLang serving Llama 4 supports quantized KV alongside prefix sharing to keep long shared system prompts cheap across batched requests.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy do keys prefer per-channel scaling while values prefer per-token scaling?

Sign in or go Pro to see the approach for this follow-up.

QWhy does KIVI keep a window of recent tokens in full precision?

Sign in or go Pro to see the approach for this follow-up.

QHow would you detect a KV-quant regression before it reaches users?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Claiming KV quant saves only memory. It also halves the bandwidth read per decode step, which is the actual bottleneck, and the freed memory converts directly into bigger batches and higher throughput.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.