Inference Optimization

193 questions

Serving LLMs at scale. KV cache, FlashAttention, speculative decoding, quantization, batching, vLLM, TensorRT-LLM, TTFT vs TPOT tradeoffs.

Sort

Questions

In benchmarks, tokens per second measures what exactly?
FlashcardEasy·Qual 4.0
Bytes, characters, tokens: which one gets billed and how do they relate?
FlashcardEasy·Qual 4.0
Throughput, for an LLM serving system: give the working definition
FlashcardEasy·Qual 4.0
What is a tensor core and why does it dominate matmul throughput on GPUs?
FlashcardEasy·Qual 4.0
How does the temperature parameter reshape the sampling distribution?
FlashcardEasy·Qual 4.0
Define a stop sequence in an LLM API call
FlashcardEasy·Qual 4.0
Server-Sent Events streaming: define it and explain why chat APIs default to it
FlashcardEasy·Qual 4.0
Sequence length vs context window, what is the practical distinction and which one drives KV cache size?
FlashcardEasy·Qual 4.0
The roofline model in plain language, without the chart, what is it saying?
FlashcardEasy·Qual 4.0
TPM and RPM each throttle a different axis, which one caps which?
Match PairsEasy·Qual 4.0
Prompt caching shows up on Anthropic and OpenAI pricing pages, what does it actually do?
FlashcardEasy·Qual 4.0
Walk through what actually happens during the prefill phase of an LLM forward pass.
FlashcardEasy·Qual 4.0
NVLink vs PCIe: match each interconnect to its bandwidth tier in a modern multi-GPU server.
Match PairsEasy·Qual 4.0
A MIG slice on an H100, what is it and what is it good for?
FlashcardEasy·Qual 4.0
What unit measures GPU memory bandwidth and why does that number cap decode speed?
FlashcardEasy·Qual 4.0
max_tokens vs stop: pair each parameter with the cap it enforces on the completion.
Match PairsEasy·Qual 4.0
Define LLM inference and explain how it differs from training.
FlashcardEasy·Qual 4.0
End-to-end latency: fill in the queue, prefill and decode components.
Fill in BlankEasy·Qual 4.0
Why do hosted LLM APIs charge separate per-million rates for input and output tokens?
FlashcardEasy·Qual 4.0
Identify the NVIDIA H100: what generation, what memory, what tensor-core formats?
FlashcardEasy·Qual 4.0
Pair each decoding strategy with its determinism and diversity profile.
Match PairsEasy·Qual 4.0
Where does FP8 fit in modern LLM serving and which GPUs support it?
FlashcardEasy·Qual 4.0
Match each 4-bit format to its niche in 2026 inference and fine-tuning.
Match PairsEasy·Qual 4.0
FLOPs vs FLOP/s: fill in the counts versus rate distinction.
Fill in BlankEasy·Qual 4.0
What is the EOS token and how does the runtime use it?
FlashcardEasy·Qual 4.0

Showing 51–75 of 193

← Prev Next →

KV Cache & LLM Serving (5 cards)

Atomic flashcards for this topic. Free for everyone. Pro syncs them into your spaced-repetition queue.

What is the KV cache?

Memory of K (key) and V (value) tensors from all prior tokens. Reused at each decoding step to avoid O(n²) recomputation.

How does KV cache memory scale?

Linearly with sequence length × layers × heads × head_dim × 2 (K and V).

What is GQA?

Grouped-Query Attention, multiple query heads share the same K and V projections, reducing KV cache size at minimal quality cost.

What is PagedAttention?

vLLM's technique, treats the KV cache as paged memory blocks, reducing fragmentation when serving many concurrent requests of varying lengths.

+ 1 more cards in the full deck · Drill all cards (Pro)

Master this topic

Daily practice mixes topics. Topic specific drill is a Pro feature.

Start practice