KV Cache

For 1M-token context decode, which parallelism axis actually relieves KV pressure?

Heavy multi-turn chat shares system prompts across requests, pick the serving framework that exploits that best.

At batch 16 and 4k context with FP8 weights, which model still fits on one 80 GB H100?

Per-token decode time grows linearly as a chat session lengthens, debug it

A teammate randomized system prompts per user and expects the prompt cache to still help, what's wrong?

Spot the error: 'we autoscale our LLM serving pods on CPU utilization.'

Walk through how a prefix-cache hit changes the work that chunked prefill has to do

Which quantization combo squeezes the most decode bandwidth per percent of quality lost?

Rank these chat-API cost levers from highest to lowest typical ROI

Memory creeps up for hours then OOMs at 3am: diagnose the KV-cache leak pattern

Name the dominant cost driver of GPT-5.5 latency at a 200k-token input

Each decode step loads ___ from HBM, regardless of how many tokens have already been generated

Pair KV-cache eviction and KV-quantization with the situation each one is the right answer for

Sequence length vs context window, what is the practical distinction and which one drives KV cache size?

Prompt caching shows up on Anthropic and OpenAI pricing pages, what does it actually do?

Walk through what actually happens during the prefill phase of an LLM forward pass.

Why do hosted LLM APIs charge separate per-million rates for input and output tokens?

Describe what happens during the decode phase of LLM inference.

What is the context window of an LLM?

Recognize vLLM and the optimization that made it famous

Identify PagedAttention and the project it ships with

Expand MQA and state its KV-cache cost tradeoff

Pair HBM and SRAM with their roles on an inference GPU

Define GQA in transformer attention

Tying K and V to one projection, what is gained and what is risked?