What is the EOS token and how does the runtime use it?

**EOS** is short for End Of Sequence. It is a special token built into the model's vocabulary, like an invisible punctuation mark that means stop here. During training, the model learns to emit EOS when the response feels complete. At inference, every decode step samples one token from the model's predicted distribution. If that token is EOS, the serving runtime stops generating immediately and returns the reply to the user. The finish reason matters for debugging. When EOS triggers the stop, the API reports finish_reason=stop. When the response hits its max-tokens budget instead, the API reports finish_reason=length, which means the model was cut off mid-thought. If the caller provided a stop string and the model output matched it, the finish reason is also stop, but the cause is the stop string, not EOS. Knowing which condition fired tells you whether the model thought it was done or whether you truncated it. The two situations call for very different fixes.

**EOS** stands for End Of Sequence and is one of several special tokens reserved in the model's vocabulary. The model is trained, both in pretraining and in instruction tuning, to emit EOS when it judges the response complete. At inference, every decode step samples from the next-token distribution. When the sampled token's id matches the EOS id (or one of several EOS ids in models with multiple), the serving runtime halts generation, strips the EOS from the visible output, and returns the response with finish_reason=stop. EOS is distinct from two other stop conditions. A max-tokens budget terminates generation regardless of what the model wanted to do; the API reports finish_reason=length. A user-supplied stop string matches against the emitted text and terminates the moment the match appears; finish_reason is still stop, but the trigger is the caller's policy rather than the model's judgement. Getting EOS right is fragile. Fine-tuning that uses chat templates inconsistently can train the model to never emit EOS at all, leading to runaway generations. Newer chat models like Llama 4 and Qwen 3.5 actually use multiple EOS-like tokens for end-of-turn versus end-of-conversation, and the serving runtime has to know all of them.