Weight quant compresses static tensors offline; KV cache quant compresses runtime activations with per-request distributions, layer compounding errors, and long context noise amplification.
Picture two shrinking jobs. The first is shrinking a textbook that never changes, you scan it once, pick the best shrink trick, ship the smaller copy, and you're done forever. The second is shrinking every live conversation as it unfolds, with no way to know what the next sentence will say, and every tiny shrinking mistake at the start ripples into how the rest of the chat gets stored. The first is gentler because the textbook is steady; the second is touchier because mistakes pile up. Halve the size by storing one byte per number instead of two and chats survive; squeeze to a quarter byte and the model starts forgetting math.
Detailed answer & concept explanation~6 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Walk the static vs dynamic distinction, the calibration asymmetry, K's outlier channel pathology, layer wise error accumulation, softmax driven long context amplification, and the production landscape (per-token/per-channel quantization, fp8 on Hopper, KIVI/KVQuant for aggressive bitwidths).
| Aspect | Weight quantization | KV cache quantization |
|---|---|---|
| What's compressed | Pretrained parameters | Runtime K, V activations |
| When | Offline, one time | Online, every request |
| Distribution known? | Yes, fixed | No, varies per request |
| Calibration | Run once with corpus | Heuristic per-token/per-channel scales |
| Error propagation | Layer by layer matmul noise | Compounds through deep attention stacks AND across decode steps |
| Long context impact | Mild | Strong, softmax aggregates more noise |
| Production INT8 | Routine | Routine |
| Production INT4 | Mature (AWQ, GPTQ) | Risky on hard tasks |
Real products, models, and research that use this idea.
- vLLM and SGLang both ship fp8 KV cache as a production option on H100/H200/B200, often paired with fp8 weights for compounding savings.
- llama.cpp and TensorRT-LLM support 4-bit and 8-bit KV; users routinely report quality drops at 4-bit on GSM8K, MATH, and long context retrieval.
- KIVI treats K per-channel and V per-token, hitting near-fp16 quality at 2-bit on Llama-class checkpoints.
- DeepSeek V4 deployments combine MLA with int8 latent quantization for cumulative cache savings well past what GQA-plus-fp16 reaches.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy are K outlier channels worse than V outlier channels?
QHow does fp8 KV cache differ from int8 KV cache?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating KV cache quantization as 'just another quantization' interchangeable with weight quantization, they have different distributional properties, different sensitivities, and different error accumulation dynamics.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
- Liu et al. 2024 — KIVI: A Tuning-Free Asymmetric 2-bit Quantization for KV Cache
- Hooper et al. 2024 — KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
- Frantar et al. 2022 — GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Lin et al. 2023 — AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
Same topic, related formats. Practice these next.