Weight INT8 is the mature, easy path; INT8 KV cache is the harder one because activations have wider dynamic range and errors compound across layers.
Picture two different freezing problems. Freezing a shelf of canned soup is easy: the cans are identical, you know their size, you stack them once and never touch them again. Freezing the day's catch on a fishing boat is harder: the fish are different sizes, some species swell when frozen, and every time you reopen the freezer the temperature shifts a little. Weights are like the canned soup, static and predictable. The KV cache is like the daily catch, changing with every input and read many times by later attention layers. Same INT8 storage size, very different difficulty.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
INT8 quantization is one of the highest leverage tools an inference engineer has, but the two big targets, weights and the KV cache, behave very differently. Conflating them is a common cause of production quality regressions.
Weights are static. After training they sit in memory unchanged, with known distributions per channel. You can spend hours offline analyzing them and engineering scales that lose almost no quality. Tooling for this (AWQ, GPTQ, SmoothQuant) is mature, and a 2026 production deployment of Llama 4 Maverick or Mistral Large 3 routinely runs INT8 or INT4 weights with near-zero perplexity hit.
The KV cache is the opposite. K and V are activations: they vary with every input, carry well known outlier channels, and are reread by every later attention step at that layer. INT8 quantization here is harder, requires per-channel or per-token scales, and ships behind FP8 KV in most production toolchains. This deep dive walks the asymmetry and the practical playbook.
Why weights are the easy quantization target
Weights have three properties that make INT8 quantization a near-solved engineering problem in 2026.
Property 1: static distribution
After training, every weight tensor's distribution is fixed. You profile once on a small calibration set, fit per-channel scales, and the result is correct for every future inference. There is no risk that tomorrow's input shifts the distribution.
Property 2: channel-stable structure
Weight matrices tend to have well behaved per-output-channel distributions. Per-channel scales (one scale per output column) handle the variation cleanly without per-row complexity. AWQ exploits this by upweighting the channels most sensitive to quantization error, GPTQ uses second-order information from the Hessian, SmoothQuant migrates activation outliers onto the weight side.
Property 3: error budget is one-shot
A rounding error in a weight value contributes to one matrix multiplication per forward pass. The output flows through one residual and one normalization, both of which absorb a lot of noise. End to end perplexity barely moves.
The result: INT8 weights are the standard default in vLLM, TensorRT-LLM, and llama.cpp serving stacks in 2026.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | INT8 weights | INT8 KV cache |
|---|---|---|
| Value type | Static parameter | Dynamic activation |
| Calibration | Offline, one-time | Holds across all serving inputs |
| Outlier handling | Migrate scale into activations (SmoothQuant) | Per-channel or per-token scales required |
| Error impact | One matmul per forward pass | Compounds across every decode step |
| 2026 tooling | AWQ, GPTQ, SmoothQuant | vLLM FP8, TensorRT-LLM INT8, llama.cpp Q8_0 |
| Production maturity | Standard default | Adopted selectively after FP8 KV |
Real products, models, and research that use this idea.
- vLLM offers `kv_cache_dtype=fp8` and `fp8_e4m3` as the first widely deployed cache compression below FP16.
- TensorRT-LLM supports INT8 KV with per-channel scales plus a calibration step on a representative prompt set.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is FP8 KV easier to adopt than INT8 KV at the same bit width?
FP8 keeps an exponent so dynamic range stays wide without per-channel scales; INT8 collapses to a single scale per group and loses the outlier-friendly representation.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming both INT8 paths share the same calibration pipeline and risk profile. KV cache values are dynamic activations with outlier channels, while weights are static and channel-stable. Plan them as separate engineering problems.
60 second bullets to scan on the way to the call.
Why weights are easier to quantize than activations in general
Activation outlier channels and the SmoothQuant insight
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.