Does Anthropic's message_start event count toward TTFT in production benchmarks?

Attempt it

Anthropic's streaming API emits a message_start SSE event before any content tokens arrive. A teammate is benchmarking TTFT against your existing OpenAI latency dashboard and proposes using the message_start timestamp as the first-token marker. Explain whether that aligns with how TTFT is conventionally defined, what message_start actually represents on the wire, and how the choice would distort a side by side comparison.

Free · 2 AI evals / day

The proposal does not align with how TTFT is conventionally defined. **TTFT** stands for Time To First Token, and the convention is that it measures wall-clock from request submission to the first **content** token: the first piece of text the user actually sees in the response. The definition is about user experience, not about wire-level events. Anthropic's streaming protocol emits several SSE events in order. The first one is `message_start`, which carries metadata: the message ID the server assigned, the model name, the role, and placeholder usage counts. No visible text. Then comes `content_block_start`, also metadata. Then one or more `content_block_delta` events with `type: text_delta`, and those are the events that carry the actual response characters. If you time TTFT against `message_start`, you are timing how fast Anthropic acknowledges the request. That happens essentially as soon as the server accepts the connection, before prefill has produced anything. The gap between `message_start` and the first `text_delta` is the real prefill window, often tens to hundreds of milliseconds depending on prompt length. OpenAI's streaming has no equivalent metadata preamble. Its first chunk is typically empty followed by the first actual text. Comparing the two protocols requires picking semantically equivalent events: first content delta on both sides.

**Short answer: no, it would distort the comparison.** TTFT is defined as time-to-first-CONTENT-token, and `message_start` is not a content token. The Anthropic SSE event sequence is: `message_start`, `content_block_start`, one or more `content_block_delta` events with `type: text_delta`, then `content_block_stop` and `message_stop`. Only the `text_delta` events carry visible characters. `message_start` is a session preamble: it announces the message ID, model name, role, and zero-initialized usage counts. The server can emit it as soon as it has accepted the request and assigned an ID, well before prefill begins to produce output. Using `message_start` as the TTFT marker measures connection latency plus admission overhead, not the user-perceived prefill window. The first text actually arrives at the first `text_delta`, which can be tens to hundreds of milliseconds later, especially for long prompts where prefill is the heavy phase. The comparison with OpenAI breaks because OpenAI's streaming protocol has no comparable metadata-only preamble. Its first chunk is typically `choices[0].delta.role` (also metadata-ish but smaller and often near-immediate), and the first chunk with non-empty `choices[0].delta.content` is the first content marker. To compare apples to apples, instrument both clients to start the timer at request submission and stop it at the first event that contributes visible characters: `content_block_delta` of type `text_delta` for Anthropic, first chunk with non-empty `delta.content` for OpenAI. Anything else is measuring different things and pretending they are the same. The practical impact: the wrong marker would systematically flatter Anthropic by the size of its prefill window, which grows with prompt length. A 200-token prompt might show a small bias; a 2000-token prompt might show a 200-millisecond bias, turning a real prefill gap into a misleading tie.

Quick reference

TL;DR

No. message_start is a metadata preamble emitted before prefill output, so timing TTFT against it understates the user-perceived wait and flatters Anthropic versus OpenAI by tens to hundreds of milliseconds.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine timing how long it takes to get food at a restaurant. You could start counting from when the waiter brings the menu, or from when the first bite of food arrives. Anthropic's message_start is the menu: it shows up almost instantly because it just announces the order, but no food has been cooked yet. OpenAI's protocol does not have a separate menu-delivery step; the waiter just walks over with the food. If you use menu-arrival as your timing marker for Anthropic but food-arrival for OpenAI, you make Anthropic look much faster than it is. The fair comparison times when food actually appears on both sides.

Key concepts

Time to first token Streaming inference Server-sent events Prefill Benchmarking

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: TTFT definition is first CONTENT token + message_start is metadata preamble + correct marker is content_block_delta with text_delta + OpenAI has no equivalent preamble + bias grows with prompt length + fair benchmark instruments both first-content events.

Real-world examples

Real products, models, and research that use this idea.

Anthropic Claude Opus 4.7 and Sonnet 4.6 emit message_start within milliseconds of connection accept, well before prefill output.
OpenAI GPT-5.5 streaming chunks start with a role-only delta, then text deltas appear after prefill.
Public benchmark suites like Artificial Analysis explicitly note that TTFT must be measured at the first content delta, not the first SSE event.
Internal SRE teams at large LLM consumers routinely catch this bug when validating provider SLOs.
Gemini 3.1 Pro's streaming protocol has its own preamble shape, so cross-provider TTFT benchmarks must instrument every protocol's first-content event individually.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you instrument both clients to make the comparison truly fair?

Sign in or go Pro to see the approach for this follow-up.

QWould you ever report message_start latency as a separate metric?

Sign in or go Pro to see the approach for this follow-up.

QWhat other measurement traps appear in cross-provider LLM benchmarks?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating any SSE event as the first-token marker. Only the first event that carries visible characters counts. Metadata preambles do not.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.