Latency

TTFT: name the metric it captures and what dominates it

Throughput, for an LLM serving system: give the working definition

End-to-end latency: fill in the queue, prefill and decode components.

Pick the workload where a provider's Batch API beats the realtime endpoint

Match TTFT and TPOT to the latency they measure during streaming

Why does an agent loop with 5 tool calls have ~6× the latency of a single completion and how do you mitigate?

Which mitigation has the largest impact on agent-loop latency when tool calls are independent?

For a 200-token prompt and 1000-token completion, why does decode dominate end-to-end latency?

Spot the errors in this 'API is slow because of network and tokenizer' explanation

How should a production MCP host handle a tool call that exceeds the latency budget?

What is the correct approach when an MCP tool call exceeds its per-tool timeout?