KV Cache

Walk through KV-cache updates when speculative decoding rejects some draft tokens

During decode, why is only Q computed for the new token while full K and V come from cache?

Prefill and decode: name the two phases of LLM inference and say which one is compute-bound

vLLM defaults to a page size of 16. Pick what makes 1 and 1024 worse choices.

Spot the GQA configuration error: num_attention_heads=32, num_key_value_heads=7.

Why does MQA underperform GQA when both share KV across heads?

MQA, spell out the acronym and name exactly what gets shared across heads

Identify the attention-pattern difference between Mistral 7B and Llama-2 7B

Across which axes is the KV cache stored, per layer, per head, per batch?

INT8 KV cache vs INT8 weight quantization, which one is easier in production?

Llama-2 70B uses 64 query heads, how many KV heads does it actually keep?

GQA, decode the acronym and describe its KV sharing pattern

Flash-Decoding parallelizes an axis FlashAttention v2 left alone. Pick which one.

Name two production KV-cache eviction policies and what each prioritizes

Translating one source sentence into many target languages, what attention-side trick scales?

Decoder-only LLMs ship without a cross-attention sub-layer, so where does the context go?

Decode-phase attention is memory bandwidth bound. Explain what flips the regime.

Describe how continuous batching interacts with the KV cache

How does prefix KV-cache sharing reduce serving cost across requests?

Name two recovery moves a serving stack can pull when KV memory runs short mid-request.

You change one word in the system prompt and prompt caching breaks. Why?

How does PagedAttention enable prefix sharing across requests and what data structure makes it cheap?

What is RadixAttention in SGLang and how does it differ from vLLM's prefix sharing?

Which workload most benefits from RadixAttention specifically vs simpler per-request prefix caching?

How do Anthropic and OpenAI prompt-cache pricing models work and what counts as a cache hit?