KV Cache

Why is KV cache memory the bottleneck for long context LLM serving?

Compute the KV cache memory for a single request at 4096 context on Llama-2 70B (MHA) in FP16.

Explain attention as content-addressable memory. Where does the analogy hold and where does it break?

In LLM serving, what is the primary driver of end to end latency for a generation request?

What is the KV cache in transformer inference?