INT8 quantization means storing numbers using 8-bit integers instead of 16-bit or 32-bit floats. It cuts memory and often speeds up math, but it loses precision, so quality may drop. The question is where INT8 is easier to apply: to the model's frozen weights, or to the KV cache that holds K and V vectors during generation.
Weight quantization is the easier and more mature path. Weights are fixed after training, so you can profile them once, find a good per-channel scale, and store the compressed version. Toolkits like AWQ, GPTQ, and SmoothQuant make this near plug and play, and the quality hit is typically tiny.
INT8 KV cache is harder. The K and V tensors are activations that depend on the input, change with every request, and grow with the sequence length. Some channels carry outlier values that naive scaling will clip. And because every later attention step reads these cached values, small rounding errors compound across the stack.
In 2026 production stacks, weight INT8 is standard; KV INT8 needs careful per-channel scales and is often deployed only after weight INT4 plus FP16 KV is exhausted.
Weight INT8 is the easy lift in 2026 production. Weights are static, have known per-channel distributions, and let you calibrate offline against a small dataset, AWQ, GPTQ, and SmoothQuant ship this near losslessly for almost every Llama 4 class model.
INT8 KV cache is the genuinely harder problem for three reasons. First, K and V are activations, so their magnitudes vary by input, by head, by position, and grow with sequence length. Per-tensor scales clip; you need per-channel or per-token scales. Second, attention activations are famous for outlier channels (the same phenomenon SmoothQuant addresses on the weight side), and naive INT8 rounding wipes them out. Third, errors compound. Cached K and V are read by every later decode step at that layer, so a small per-token rounding error accumulates across the whole generation rather than affecting one matmul.
The production order most teams follow is weight INT8 first, then weight INT4 with FP16 KV, then finally INT8 KV with grouped or per-channel scales. Tools like vLLM's `kv_cache_dtype=fp8` and TensorRT-LLM's KV calibration paths are catching up, but the engineering bar is still higher than weights.
Quick reference
TL;DR
Weight INT8 is the mature, easy path; INT8 KV cache is the harder one because activations have wider dynamic range and errors compound across layers.
Picture two different freezing problems. Freezing a shelf of canned soup is easy: the cans are identical, you know their size, you stack them once and never touch them again. Freezing the day's catch on a fishing boat is harder: the fish are different sizes, some species swell when frozen, and every time you reopen the freezer the temperature shifts a little. Weights are like the canned soup, static and predictable. The KV cache is like the daily catch, changing with every input and read many times by later attention layers. Same INT8 storage size, very different difficulty.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Latency-critical low-batch serving where the cache already fits in HBM INT8 KV adds dequant overhead on every read for no memory win when batch size is small and contexts are short.
Quality-sensitive workloads such as reasoning chains or long-form generation INT8 KV error compounds across decode steps and shows up first in the kinds of multi-step reasoning these workloads stress.
Architectures already using MLA, such as DeepSeek V4 MLA compresses the cache into a latent space at FP16; layering INT8 on top adds risk without much marginal memory savings.
Mixed-tenant serving with very different prompt distributions A single per-channel calibration may not hold across all tenants' activation patterns, leading to per-request quality variance.
Quick scan · 2–4 min
Everything important, quickly.
5m: why weights are easier than activations, outlier channels and SmoothQuant, per-channel vs per-token KV scales, error compounding across decode steps, and the 2026 production order from weight INT4 with FP16 KV through FP8 to INT8.
Side-by-side
Property
INT8 weights
INT8 KV cache
Value type
Static parameter
Dynamic activation
Calibration
Offline, one-time
Holds across all serving inputs
Outlier handling
Migrate scale into activations (SmoothQuant)
Per-channel or per-token scales required
Error impact
One matmul per forward pass
Compounds across every decode step
2026 tooling
AWQ, GPTQ, SmoothQuant
vLLM FP8, TensorRT-LLM INT8, llama.cpp Q8_0
Production maturity
Standard default
Adopted selectively after FP8 KV
Real-world examples
Real products, models, and research that use this idea.
1vLLM offers `kv_cache_dtype=fp8` and `fp8_e4m3` as the first widely deployed cache compression below FP16.
2TensorRT-LLM supports INT8 KV with per-channel scales plus a calibration step on a representative prompt set.
3llama.cpp ships `Q8_0` and `Q4_0` KV cache formats with chunked dequant kernels for CPU and Metal inference.
4Llama 4 Maverick and Mistral Large 3 deployments commonly run INT4 weights with FP16 or FP8 KV in production.
5DeepSeek V4 uses Multi-head Latent Attention to compress KV into a latent space at FP16, reducing the need to quantize per element.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is FP8 KV easier to adopt than INT8 KV at the same bit width?
Sign in or go Pro to see the approach for this follow-up.
QHow does MLA in DeepSeek V4 change the quantization calculus for the cache?
Sign in or go Pro to see the approach for this follow-up.
QWhat calibration data should you use for KV cache quantization?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming both INT8 paths share the same calibration pipeline and risk profile. KV cache values are dynamic activations with outlier channels, while weights are static and channel-stable. Plan them as separate engineering problems.