Describe how to record time to first token on a streaming LLM call
Capture two timestamps (stream open, first delta), record the delta as a numeric attribute for charting and as a span event for timeline visualization, and keep the span open until the stream closes so total latency
Picture timing a sprinter. You start a stopwatch when the gun fires (the stream opens) and a second one when their first foot crosses the start line proper (the first token arrives). The gap between gun and first step is the reaction time. Total race time is gun to finish line. You need both numbers because a fast finish with a slow reaction is a different problem from a slow finish with a fast reaction. A streaming LLM call works the same way. Time to first token is the reaction time; total stream duration is the race time. Both go on the same span so you can see them together later.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Time to first token is the latency users actually feel. In a chat UI, a 200 ms TTFT with 5 second total generation feels fast; a 3 second TTFT with 5 second total generation feels broken even though the totals are similar. The instrumentation has to capture TTFT cleanly enough that the production dashboards can alert on it and the trace UI can show it as a marker on the span timeline.
The pattern is not complicated, but the details matter. Pick the wrong start timestamp and you under-measure user-perceived latency. Pick the wrong end timestamp and you confuse empty role chunks with real content. Close the span too early and you lose end of stream timing. The sections below walk through the choices.
The two timestamps and what they include
Capture t0 immediately before the network call that opens the stream. Practically that means right before the SDK's client.messages.stream(...) or client.chat.completions.create(..., stream=True) invocation.
This choice matters. Capturing t0 after the SDK has serialized the request misses the serialization time. Capturing it after the TCP connection is established (some SDKs expose this) misses connection-setup time. Both of those are part of what the user waited for, so neither is the right choice for a user-perceived metric. Capture t0 at the syscall boundary as close to 'now we are starting the request' as the SDK allows.
Capture t1 on receipt of the first delta that contains actual content. Some providers emit empty role-only first chunks; treating those as the first token causes TTFT to swing wildly between providers and across SDK versions. Gate the t1 capture on the first chunk with a non-empty content field:
- OpenAI: first
choices[0].delta.contentthat is a non-empty string. - Anthropic: first
content_block_deltaevent with a non-emptytext(orpartial_jsonfor structured output). - Generic OTel pattern: first chunk that contributes a token to the output.
The difference between the right gate and the wrong gate is typically 50-150 ms; small, but enough to make TTFT comparisons across providers meaningless.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenTelemetry GenAI semantic conventions in 2026 define `gen_ai.response.time_to_first_token_ms` as a recommended attribute on streaming spans.
- Anthropic's Claude Opus 4.7 streaming API emits an initial `message_start` event followed by `content_block_delta` events; the first content_block_delta is what triggers t1.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you instrument inter-token latency without exploding the events per span count?
Sample. Emit a token-arrival event every Nth token (e.g. every 32nd) so you can chart inter-token latency without recording 500 events for a long generation. For high-fidelity debug traces, flip the sampling rate to 1 on a small percentage of traffic.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Closing the span as soon as the first token arrives, treating TTFT as the whole latency. You lose end of stream timing and cannot separate prefill from decode latency.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.