EOS is the stop signal. The model learns to emit it when a response is complete; the generation loop watches for it and halts. If training labels never include EOS, for instance because a chat template change stripped it before the loss was computed, the model never learns to stop. At inference it runs until max_new_tokens, padding completions with rambling continuations or random follow-up turns.
**EOS** stands for end of sequence. It is a single special token in the model's vocabulary that means 'this response is complete.'
During training, EOS is the very last target token in every example. The model sees a question and a response, and at the position right after the last word of the response, the target is EOS. The cross-entropy loss pushes the model to assign high probability to EOS at exactly that position, learning that EOS belongs at the end of a complete answer.
At inference, generation runs as a loop: sample the next token, append it to the running sequence, feed it back in, repeat. The loop has one natural exit condition: if the sampled token is EOS, stop. The generated text up to but not including EOS is returned as the response.
If training labels never contain EOS, the model never learns to assign high probability to it. The token is technically in the vocabulary but the model treats it as essentially never the right answer. At inference the generation loop keeps sampling other tokens forever, or until it hits the `max_new_tokens` hard cap. The output ends with rambling continuations, made-up follow-up turns, or repetitive padding text.
The most common cause of this bug is a chat template change that strips EOS from the training labels, often during a custom preprocessing step.
**EOS is the stop signal**, both at training time and inference time, and the entire generation control flow depends on the model having learned to emit it.
During supervised fine-tuning the EOS token sits at the end of every assistant response in the training labels. The cross-entropy loss pushes the model to assign high probability to EOS at that specific position, given everything that came before. By the end of training the model has gradient-aligned EOS to the meaning 'I am done answering.'
At inference the generation loop has a straightforward exit condition. After sampling each new token, it checks whether the token equals the model's EOS ID (or any of a configurable list of stop tokens). If yes, stop and return. If no, append and continue. The loop also has a hard cap on `max_new_tokens` to prevent infinite runaway, but in normal operation EOS is what triggers stopping.
If EOS never appears in the training labels, the model effectively learns that EOS has zero probability everywhere. The generation loop runs until `max_new_tokens`, at which point it truncates whatever the model was doing. The truncated text typically looks like a complete response followed by random rambling: the model produces a reasonable answer, then continues with hallucinated next-turn user messages, repetitive padding, or wandering tangents because nothing in its training signaled where to stop.
The most common root cause is a chat template that strips EOS during preprocessing, often because the template change happened after the fine-tuning recipe was authored and nobody re-verified the rendered output. The fix is to render a sample example end to end and visually confirm the EOS token is present at the close of every assistant span.
Quick reference
TL;DR
EOS is the stop signal. If training labels never include it, the model never learns to emit it and generation runs until max_new_tokens at inference.
Imagine teaching a child to read aloud, but you never show them what a period looks like or what it means. They will read each sentence into the next sentence without pausing, because nobody ever taught them where a thought is supposed to stop. The EOS token is the period for a language model. The model learns from examples to put it at the end of a complete response, and the generation system watches for it like a listener watching for a pause. If the training data has the period stripped out, the model fluently produces words forever, because it has no idea that stopping is even an option. The fix is always to put the period back into the training labels.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Base completion model fine-tuning where there is no notion of response boundary Continuous text completion has no natural stop signal; EOS handling does not apply because the model is not expected to terminate at a specific point.
Streaming generation that uses external stop sequences instead of model-emitted EOS If your serving stack stops on regex matches against the rendered text, the model does not need to emit EOS reliably; the stop logic lives at the application layer.
Fine-tuning for token-level classification rather than generation Classification heads do not run a generation loop, so EOS has no role in the downstream task even though the underlying model has the token in vocabulary.
Quick scan · 2–4 min
Everything important, quickly.
4 min: define EOS as the stop signal, walk through the training-time and inference-time mechanics, then enumerate the three failure modes (template strip, tokeniser swap, multi-stop mismatch) and the sanity check that catches each.
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face TRL's SFTTrainer warns when training examples lack an EOS token at the end of the assistant turn, exactly because the failure mode is so common.
2Llama 4 Maverick fine-tunes use `<|eot_id|>` as the end of turn stop token, which must be both in training labels and in the inference stop_token list.
3Qwen 3.5 fine-tunes that mix chat templates often drop `<|im_end|>` from labels, producing post-deployment models that ramble past the assistant turn.
4DeepSeek V4 distillation runs explicitly include eos_token in the loss-computed span of every example to avoid the never-stops failure mode.
5Production support requests at Hugging Face and Anthropic frequently cite 'model never stops generating' as a top-five fine-tuning bug, traced back to EOS handling almost every time.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf the model emits a stop token but the generation loop does not halt, where is the bug?
Sign in or go Pro to see the approach for this follow-up.
QHow does loss masking interact with EOS, and why is this often the silent root cause?
Sign in or go Pro to see the approach for this follow-up.
QWhy does a model with no EOS gradient often produce a believable response followed by rambling, rather than rambling from the start?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Stripping EOS during dataset preprocessing because it 'looks like padding' and then wondering why the fine-tuned model rambles forever past the end of its response.