Contrast time to first token with tokens per second as serving SLOs

Flashcard·Medium·4.0 · 0·~30s·Asked atDust Hebbia Snap

Attempt it

An interactive chat app cares about time to first token because the user's perceived latency is the gap between sending and seeing the first visible character. Past that point, streaming reduces apparent latency even if total generation takes longer. So an interactive UI needs TTFT under 500ms ideally, with the trailing tokens per second only needing to be fast enough that the stream feels alive (15-30 tok/s is comfortable).

A batch summarizer has no user watching the stream. The thing that matters is wall-clock time to complete the whole output, which is dominated by tokens per second for any output longer than a few tokens. TTFT is irrelevant; you optimize for throughput per GPU and per dollar.

The serving runtime configuration follows: continuous batching, large batch sizes, and quantization help throughput; smaller batches and prefill optimization help TTFT. Most production systems run separate fleets when both workloads are heavy.

Your answer

LLM serving SLOs split along two axes that measure different parts of the request lifecycle. **Time to first token (TTFT)** is the latency from sending the request to seeing the first generated character. It is dominated by **prefill**: the model has to process the entire input prompt before it can produce the first output token. For a chat application, this is the user-perceived latency, the gap between hitting enter and seeing the response start. Streaming the rest of the output makes the total generation time matter less, as long as the stream feels alive. **Tokens-per-second (TPS)** is the decode throughput: how fast the model emits tokens after the first one. It dominates wall-clock time for any output longer than a few tokens. For batch workloads with no user watching the stream, TPS is what matters; for interactive workloads, TPS only needs to keep up with reading speed (15-30 tok/s is comfortable). **Workload mapping.** An interactive chat app needs TTFT under 500ms ideally and TPS fast enough to stream readably. A batch summarizer or a data-extraction pipeline does not care about TTFT at all; the SLO is throughput-per-GPU and cost per million tokens. **The serving config trade-off.** Continuous batching, large batch sizes, and quantization help TPS but can hurt TTFT (a new request waits for the next batch slot). Smaller batches and prefill optimization help TTFT but reduce TPS. Production systems with both workloads usually run separate fleets.

Inference SLOs split along two metrics that measure different parts of the request lifecycle, and the workload determines which one matters. **Time to first token (TTFT)** measures the gap from request submission to the first emitted token. It is dominated by prefill cost: the model must process the entire input prompt before it can start generating. For a 2,000-token input, prefill is a meaningful fraction of total request time. TTFT is the latency the user feels in an interactive surface; once tokens start streaming, perceived latency drops dramatically even if total generation continues. **Tokens-per-second (TPS)** measures decode throughput after the first token. It dominates wall-clock time for any output longer than a few tokens and is the metric that drives cost per million tokens for self-hosted serving. **Workload mapping.** An interactive chat needs TTFT under 500 ms for a good experience and TPS just fast enough to stream readably (15-30 tok/s). A batch summarizer or extraction pipeline does not care about TTFT; throughput-per-GPU dominates the economics. A voice agent needs both: sub-100 ms TTFT for natural turn-taking and high enough TPS to keep the speech synthesis fed. **Serving-config trade-off.** Continuous batching with large batch sizes maximizes TPS by keeping the GPU saturated, but a new request can wait for a batch slot, hurting TTFT. Quantization (FP8, AWQ) helps TPS by reducing memory bandwidth pressure. Prefill optimization (chunked prefill, prefix caching) helps TTFT by reducing time to decode start. Smaller batches and lower concurrency improve TTFT at the cost of throughput. **The production pattern.** When both workloads are heavy, run two fleets. Interactive fleet: small batches, prefill optimization, TTFT-tuned config. Batch fleet: large continuous batches, aggressive quantization, throughput-tuned config. Sharing one fleet for both workloads usually leaves performance on the table on both metrics.

TL;DR

TTFT is the user-perceived latency floor for interactive surfaces, dominated by prefill; tokens per second is the wall-clock metric for non-interactive workloads, dominated by decode batching.

Memory aid

Easy to grasp

Picture two diners at the same restaurant. One is hungry and just wants the bread to land at the table so they know food is coming. They will happily wait for the rest as long as something arrives quickly. The other is at a takeout window placing a big catering order; they do not care when the first dish is ready, only when the whole order is bagged and they can leave. The hungry diner cares about how fast the bread shows up (time to first token); the takeout customer cares about total prep time (tokens per second over the whole order). The kitchen configures itself differently for each: small fast batches for the bread crowd, large efficient batches for the takeout.

Key concepts

LLMOps Vector databases

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

LLM serving SLOs split along two metrics that look similar at first glance but measure different parts of the request lifecycle and respond to different optimizations. Confusing them leads to serving configurations that fail one SLO while optimizing the other, often without the operator realizing why. The TTFT versus TPS distinction is foundational for any production LLM serving decision in 2026.

This card walks through what each metric measures, what dominates it, how the serving-runtime configuration affects the trade-off, and why mature deployments often run separate fleets for interactive and batch workloads.

TTFT: the prefill-dominated metric

Time to first token measures the gap from request submission to the first emitted output token. The components are queue wait, prefill (forward pass over the full input prompt to populate the KV cache), and scheduling overhead.

For typical interactive inputs (1k-4k tokens including system prompt) prefill is a meaningful fraction of total request latency, often the dominant component. Larger inputs (8k+ tokens in long-context RAG or agent loops) make prefill the overwhelming TTFT cost.

The user-perception story. In an interactive surface, the user is staring at the screen between submitting and seeing the first character. That gap is the perceived latency of the system, regardless of how fast tokens stream after. A 2-second TTFT followed by 50 tok/s streaming feels slow; a 300 ms TTFT followed by 20 tok/s streaming feels snappy. Streaming hides total generation time from the user; it cannot hide TTFT.

Optimizations that help TTFT. Chunked prefill interleaves prefill and decode work so a new request can start streaming its first tokens before its full prefill has finished, lowering effective TTFT under concurrency. Prefix caching (RadixAttention-style sharing of KV across requests with shared prefixes) eliminates redundant prefill on the shared portion, which is huge for agent workloads with long system prompts. Smaller batch sizes reduce queue wait. Faster prefill kernels (FlashAttention v3) cut the raw prefill compute time.

TPS: the decode-dominated metric

The inverse correlation and the trade-off

The two-fleet production pattern and SLO targets

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

ChatGPT's interactive surface runs serving configurations tuned for sub-500 ms TTFT, with chunked prefill and prefix caching to absorb long system prompts.
Anthropic's batch API trades TTFT for half-price token costs, optimized for throughput-per-GPU on workloads where latency does not matter.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you choose batch size for an interactive fleet?

Start small (4-8), measure TTFT P95 and TPS P50 under representative concurrency, increase until TTFT begins violating SLO, back off one step; chunked prefill widens the safe range.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Tuning one fleet for both SLOs. Large continuous batches help throughput but hurt TTFT; the production answer is usually two fleets with different configurations when both workloads are heavy.

60 second bullets to scan on the way to the call.

TTFT as prefill-dominated, user-perceived latency for interactive surfaces
TPS as decode-dominated, wall-clock metric for batch workloads

Primary sources. Browse if you want the original framing.