How does DeepSeek's Multi-Latent Attention (MLA) compress the KV cache below GQA?

The KV cache stores keys and values for every past token so decoding does not recompute them. Its size scales with the number of KV heads times head dimension, per layer, per token. At long context it can dwarf the model weights. Grouped-Query Attention shrinks this by sharing K and V across groups of query heads. Fewer KV heads means a smaller cache. But this is a structural constraint: you simply have fewer distinct keys and values to store. **Multi-head Latent Attention** takes a different route. It learns a small shared latent vector per token, often around 512 numbers, and caches only that. The full per-head keys and values are not stored at all. At attention time, the model expands that latent back into per-head K and V using learned up-projection matrices. Because those matrices can be folded into the existing query and output projections, there is no separate decompression step bolted on. The payoff: the cache drops by roughly ten times versus full attention, and quality holds because the latent is trained end to end. The cost is a small extra matmul, easily hidden behind the memory bandwidth you save.

**Multi-head Latent Attention** caches a single low-rank latent vector per token rather than full per-head keys and values. The latent dimension is small, around 512 in DeepSeek's models, versus the much larger product of KV heads and head dimension. At attention time the model reconstructs per-head K and V from that latent. It multiplies the latent by learned up-projection matrices. Crucially these matrices fold into the query and output projections. So the reconstruction fuses into the existing attention math with no explicit decompression pass. The cache shrinks by roughly the ratio of full KV width to latent width. That is about ten times smaller than multi-head attention. It is roughly three to five times smaller than Grouped-Query Attention with eight KV heads. Quality matches or beats GQA. The reason is the latent is learned end to end. GQA imposes a rigid structural share across head groups. MLA instead learns which correlations to keep. Runtime cost is a small extra matmul for the absorbed up-projection. Decode is bandwidth-bound, so that compute hides behind the bandwidth saved on smaller KV reads. DeepSeek V2 and V3 ship MLA in production.