Serving

Memory creeps up for hours then OOMs at 3am: diagnose the KV-cache leak pattern

Throughput, for an LLM serving system: give the working definition

Define a stop sequence in an LLM API call

Server-Sent Events streaming: define it and explain why chat APIs default to it

TPM and RPM each throttle a different axis, which one caps which?

Walk through the shape of an OpenAI-compatible chat completions request and response.

Which components dominate cold-start latency when a serving replica spins up?

Trace how one slow request stalls every co-batched generation step.

Why might a LoRA that passes every eval check then garble one in five prod requests?

Describe how continuous batching interacts with the KV cache

What problem does chunked prefill solve in production LLM serving?

Merge LoRA into the base vs keep it as a swappable adapter: when do you pick which?

How does the serving stack actually decide when to stop generation and what bugs come from mis-set stop sequences?