Both **weight quantization** and KV cache quantization compress tensors from higher precision (fp16) to lower (int8, fp8, int4) to save memory. The difference is what gets compressed and when.
Weight quantization targets the model's pretrained parameters. Once training is done, weights don't change. The workflow runs offline once: calibrate on a corpus, find optimal per-channel scales using GPTQ, AWQ, or SmoothQuant, save the quantized checkpoint, ship. Distributions are stable and known; the result is a static artifact.
KV cache quantization compresses K and V tensors computed at inference time from each request's specific tokens. These tensors don't exist before the request arrives, you can't run GPTQ on something not yet computed.
Three structural challenges. First, distributions are unknown per request; large K outliers saturate int8 range. Second, errors compound across 80+ layers. Third, long context amplifies: softmax over 8k positions sums more noisy contributions.
In 2026 production, INT8 KV is reliable across vLLM, TRT-LLM, SGLang, LMDeploy. FP8 (E4M3) is the rising Hopper plus standard. INT4 KV degrades on math and long context reasoning.
Two different jobs.
**Weight quantization** (GPTQ, AWQ, SmoothQuant) compresses fixed pretrained tensors offline. Calibrate once on a corpus, find optimal per-channel scales, save the quantized checkpoint, ship. The deployed artifact is static.
KV cache quantization compresses runtime activations whose distributions aren't known in advance. Each request brings different K and V values; errors compound through 80+ attention layers; long context amplifies softmax aggregated noise. Calibration is approximate at best.
In 2026 production, INT8 KV is reliable across vLLM, TRT-LLM, SGLang, and LMDeploy. FP8 is the Hopper and newer standard with native tensor core support. INT4 ships but degrades on GSM8K, MATH, and long context retrieval. K is typically harder to quantize than V because of outlier channels, KIVI's per-channel K and per-token V split is the dominant research pattern.
Quick reference
TL;DR
Weight quant compresses static tensors offline; KV cache quant compresses runtime activations with per-request distributions, layer compounding errors, and long context noise amplification.
Picture two shrinking jobs. The first is shrinking a textbook that never changes, you scan it once, pick the best shrink trick, ship the smaller copy, and you're done forever. The second is shrinking every live conversation as it unfolds, with no way to know what the next sentence will say, and every tiny shrinking mistake at the start ripples into how the rest of the chat gets stored. The first is gentler because the textbook is steady; the second is touchier because mistakes pile up. Halve the size by storing one byte per number instead of two and chats survive; squeeze to a quarter byte and the model starts forgetting math.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Hard math or long context reasoning workloads at aggressive int4 KV Per-layer noise compounds across 80-plus layers; GSM8K-style tasks regress measurably below int8.
Pre-Hopper GPUs without native fp8 tensor cores Software fp8 emulation costs more than it saves; int8 with per-token scaling is the better default.
Already memory rich deployments at small batch and short context The cache footprint isn't binding, so the quality risk buys nothing; keep fp16 and skip the calibration risk.
MLA-based models such as DeepSeek V4 The cache is already a small latent vector; further quantization yields little memory and high relative error.
Speculative decoding draft model caches that must match the target Quantization noise breaks acceptance rate calibration between draft and target models, hurting net throughput.
Quick scan · 2–4 min
Everything important, quickly.
Walk the static vs dynamic distinction, the calibration asymmetry, K's outlier channel pathology, layer wise error accumulation, softmax driven long context amplification, and the production landscape (per-token/per-channel quantization, fp8 on Hopper, KIVI/KVQuant for aggressive bitwidths).
Side-by-side
Aspect
Weight quantization
KV cache quantization
What's compressed
Pretrained parameters
Runtime K, V activations
When
Offline, one time
Online, every request
Distribution known?
Yes, fixed
No, varies per request
Calibration
Run once with corpus
Heuristic per-token/per-channel scales
Error propagation
Layer by layer matmul noise
Compounds through deep attention stacks AND across decode steps
Long context impact
Mild
Strong, softmax aggregates more noise
Production INT8
Routine
Routine
Production INT4
Mature (AWQ, GPTQ)
Risky on hard tasks
Real-world examples
Real products, models, and research that use this idea.
1vLLM and SGLang both ship fp8 KV cache as a production option on H100/H200/B200, often paired with fp8 weights for compounding savings.
2llama.cpp and TensorRT-LLM support 4-bit and 8-bit KV; users routinely report quality drops at 4-bit on GSM8K, MATH, and long context retrieval.
3KIVI treats K per-channel and V per-token, hitting near-fp16 quality at 2-bit on Llama-class checkpoints.
4DeepSeek V4 deployments combine MLA with int8 latent quantization for cumulative cache savings well past what GQA-plus-fp16 reaches.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy are K outlier channels worse than V outlier channels?
Sign in or go Pro to see the approach for this follow-up.
QHow does fp8 KV cache differ from int8 KV cache?
Sign in or go Pro to see the approach for this follow-up.
QWhat's the interaction between KV cache quantization and FlashAttention?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating KV cache quantization as 'just another quantization' interchangeable with weight quantization, they have different distributional properties, different sensitivities, and different error accumulation dynamics.