Weight INT8 is the mature, easy path; INT8 KV cache is the harder one because activations have wider dynamic range and errors compound across layers.
Picture two different freezing problems. Freezing a shelf of canned soup is easy: the cans are identical, you know their size, you stack them once and never touch them again. Freezing the day's catch on a fishing boat is harder: the fish are different sizes, some species swell when frozen, and every time you reopen the freezer the temperature shifts a little. Weights are like the canned soup, static and predictable. The KV cache is like the daily catch, changing with every input and read many times by later attention layers. Same INT8 storage size, very different difficulty.
Detailed answer & concept explanation~5 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5m: why weights are easier than activations, outlier channels and SmoothQuant, per-channel vs per-token KV scales, error compounding across decode steps, and the 2026 production order from weight INT4 with FP16 KV through FP8 to INT8.
| Property | INT8 weights | INT8 KV cache |
|---|---|---|
| Value type | Static parameter | Dynamic activation |
| Calibration | Offline, one-time | Holds across all serving inputs |
| Outlier handling | Migrate scale into activations (SmoothQuant) | Per-channel or per-token scales required |
| Error impact | One matmul per forward pass | Compounds across every decode step |
| 2026 tooling | AWQ, GPTQ, SmoothQuant | vLLM FP8, TensorRT-LLM INT8, llama.cpp Q8_0 |
| Production maturity | Standard default | Adopted selectively after FP8 KV |
Real products, models, and research that use this idea.
- vLLM offers `kv_cache_dtype=fp8` and `fp8_e4m3` as the first widely deployed cache compression below FP16.
- TensorRT-LLM supports INT8 KV with per-channel scales plus a calibration step on a representative prompt set.
- llama.cpp ships `Q8_0` and `Q4_0` KV cache formats with chunked dequant kernels for CPU and Metal inference.
- Llama 4 Maverick and Mistral Large 3 deployments commonly run INT4 weights with FP16 or FP8 KV in production.
- DeepSeek V4 uses Multi-head Latent Attention to compress KV into a latent space at FP16, reducing the need to quantize per element.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is FP8 KV easier to adopt than INT8 KV at the same bit width?
QHow does MLA in DeepSeek V4 change the quantization calculus for the cache?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Assuming both INT8 paths share the same calibration pipeline and risk profile. KV cache values are dynamic activations with outlier channels, while weights are static and channel-stable. Plan them as separate engineering problems.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.