An engineer says 'FP8 KV cache just halves memory, no other benefit.' Which response is most correct?

Attempt it

Quick reference

TL;DR

FP8 KV cache halves memory AND halves the per-step KV read, so decode latency drops and batch size roughly doubles. It is not a memory-only win.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a librarian who must reread an entire shelf of notes before writing each new line of a story. The notes are the KV cache, and rereading them is what slows each line down. Now suppose you rewrite every note in half-size shorthand. The shelf takes half the space, which is the obvious win. But there is a second, bigger win: rereading the whole shelf now takes half the time, because there is half as much ink to scan. So every new line comes out faster too, not just stored smaller. And because each story now uses half a shelf, the same room fits twice as many stories at once. People who only mention the space savings miss the speed and capacity gains hiding in the same change.

Key concepts

KV cache quantization FP8 / INT4 KVBandwidth-bound decodeRoofline model

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: why decode is bandwidth-bound + KV as dominant byte source + dtype halving cuts read and footprint + batch and throughput compounding + KV vs weight quant + FP8 vs INT4 accuracy.

Real-world examples

Real products, models, and research that use this idea.

vLLM exposes an fp8 KV cache flag with per-token scale calibration, near-lossless on Llama 4 and used widely in 2026 production.
TensorRT-LLM (NVIDIA) ships FP8 KV cache fused into its attention kernel on H100 and B200, dequantizing without a separate pass.
DeepSeek V4 pairs Multi-head Latent Attention with low-bit KV storage to push 128k-context serving cost down further.
Llama 4 serving stacks combine GQA with fp8 KV so long-context decode stays bandwidth-feasible on a single node.
INT4 KV experiments in SGLang show throughput gains but require eval gating because long-context quality can regress.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does FP8 KV improve latency and not just memory?

Sign in or go Pro to see the approach for this follow-up.

QHow does KV cache quantization differ from weight quantization?

Sign in or go Pro to see the approach for this follow-up.

QWhen does INT4 KV become risky compared to FP8 KV?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating FP8 KV as a memory-only optimization. Decode is bandwidth-bound, so halving the KV dtype also halves the per-step read and speeds up decode latency.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.