Does Anthropic's message_start event count toward TTFT in production benchmarks?

The proposal does not align with how TTFT is conventionally defined. **TTFT** stands for Time To First Token, and the convention is that it measures wall-clock from request submission to the first **content** token: the first piece of text the user actually sees in the response. The definition is about user experience, not about wire-level events. Anthropic's streaming protocol emits several SSE events in order. The first one is `message_start`, which carries metadata: the message ID the server assigned, the model name, the role, and placeholder usage counts. No visible text. Then comes `content_block_start`, also metadata. Then one or more `content_block_delta` events with `type: text_delta`, and those are the events that carry the actual response characters. If you time TTFT against `message_start`, you are timing how fast Anthropic acknowledges the request. That happens essentially as soon as the server accepts the connection, before prefill has produced anything. The gap between `message_start` and the first `text_delta` is the real prefill window, often tens to hundreds of milliseconds depending on prompt length. OpenAI's streaming has no equivalent metadata preamble. Its first chunk is typically empty followed by the first actual text. Comparing the two protocols requires picking semantically equivalent events: first content delta on both sides.

**Short answer: no, it would distort the comparison.** TTFT is defined as time-to-first-CONTENT-token, and `message_start` is not a content token. The Anthropic SSE event sequence is: `message_start`, `content_block_start`, one or more `content_block_delta` events with `type: text_delta`, then `content_block_stop` and `message_stop`. Only the `text_delta` events carry visible characters. `message_start` is a session preamble: it announces the message ID, model name, role, and zero-initialized usage counts. The server can emit it as soon as it has accepted the request and assigned an ID, well before prefill begins to produce output. Using `message_start` as the TTFT marker measures connection latency plus admission overhead, not the user-perceived prefill window. The first text actually arrives at the first `text_delta`, which can be tens to hundreds of milliseconds later, especially for long prompts where prefill is the heavy phase. The comparison with OpenAI breaks because OpenAI's streaming protocol has no comparable metadata-only preamble. Its first chunk is typically `choices[0].delta.role` (also metadata-ish but smaller and often near-immediate), and the first chunk with non-empty `choices[0].delta.content` is the first content marker. To compare apples to apples, instrument both clients to start the timer at request submission and stop it at the first event that contributes visible characters: `content_block_delta` of type `text_delta` for Anthropic, first chunk with non-empty `delta.content` for OpenAI. Anything else is measuring different things and pretending they are the same. The practical impact: the wrong marker would systematically flatter Anthropic by the size of its prefill window, which grows with prompt length. A 200-token prompt might show a small bias; a 2000-token prompt might show a 200-millisecond bias, turning a real prefill gap into a misleading tie.