EOS is a special vocabulary token the model is trained to emit when its answer is complete; the runtime sees it, halts generation, and reports finish_reason=stop.
Picture someone telling a story out loud. They need a way to signal they are done so the listener does not just wait forever. They might say the word period or just go quiet at a clear ending point. The model has the same problem. It writes one word at a time and needs a signal that means stop here. During training it learns one special invisible word for exactly that purpose. The serving system watches the stream of tokens, and the moment that special stop word appears, it ends the reply right away. The word never gets shown to the user. Without that signal the model would keep generating until some other limit kicked in, which would feel sloppy.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
EOS is one of the smallest pieces of an LLM's vocabulary and one of the most operationally important. Every reply you have ever received from a chat model ended either because EOS fired or because some external policy clipped the output. The two cases look identical in the user-visible text but have very different implications.
This deep dive covers what EOS actually is at the token level, how the serving runtime uses it to terminate generation, how it relates to max-tokens caps and user-supplied stop sequences, why modern chat models have multiple end tokens, and how fine-tuning often breaks EOS behaviour in subtle ways.
The payoff for understanding this well is debuggability. When a deployed model produces runaway outputs, or stops mid-sentence, or generates into the next conversational turn, the cause is almost always something specific in the EOS path. Once you can read finish_reason, count EOS tokens, and trace them through the chat template, you can localise these bugs in minutes instead of hours.
The rest of this section walks through the token-level mechanics, the API-level taxonomy, the multi-EOS reality of current chat models, the common fine-tuning failure modes, and a worked example tracing a single completion through the stop logic.
What EOS is at the token level
Every tokenizer reserves a small set of special tokens with fixed ids: typically <unk>, <pad>, <bos> (beginning of sequence), and <eos> (end of sequence). The exact names vary by model family, but the role is the same. EOS is a single token id that the model is trained to predict at points where a document or response naturally ends.
During pretraining, the EOS token sits at the end of every document in the training corpus. The model learns the distribution of tokens that precede EOS, and by the loss function it learns to emit EOS when the trailing context looks like the end of a coherent unit.
At inference, EOS is just another token id that the sampler can pick. The serving runtime maintains a list of stop ids. After each decode step, it checks the sampled id against the list. If it matches, generation stops, the EOS token is removed from the visible output, and finish_reason=stop is reported to the caller.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 4 Maverick uses `<|eot_id|>` and `<|end_of_text|>` as role-specific EOS tokens, both registered as stop ids in vLLM and TGI.
- Qwen 3.5 chat models stop on `<|im_end|>`, which marks the end of each assistant turn in the ChatML format.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy do modern chat models use multiple end tokens instead of just one EOS?
Walk through the role structure of a chat conversation: system, user, assistant turns. The model needs to end its own turn without ending the whole conversation. Two tokens, end-of-turn and end-of-text, let the model signal a turn boundary that the system can resume from while still having a hard end marker for the full session.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing EOS with a max-tokens limit or a user-supplied stop string. Only EOS reflects the model's own decision; the other two are external policy.
60 second bullets to scan on the way to the call.
Define EOS as a vocabulary token.
Describe how the runtime checks sampled token ids against the EOS id.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.