Why does dropping the EOS token from SFT labels produce a model that never stops generating?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

EOS is the stop signal. The model learns to emit it when a response is complete; the generation loop watches for it and halts. If training labels never include EOS, for instance because a chat template change stripped it before the loss was computed, the model never learns to stop. At inference it runs until max_new_tokens, padding completions with rambling continuations or random follow-up turns.

Your answer

**EOS** stands for end of sequence. It is a single special token in the model's vocabulary that means 'this response is complete.' During training, EOS is the very last target token in every example. The model sees a question and a response, and at the position right after the last word of the response, the target is EOS. The cross-entropy loss pushes the model to assign high probability to EOS at exactly that position, learning that EOS belongs at the end of a complete answer. At inference, generation runs as a loop: sample the next token, append it to the running sequence, feed it back in, repeat. The loop has one natural exit condition: if the sampled token is EOS, stop. The generated text up to but not including EOS is returned as the response. If training labels never contain EOS, the model never learns to assign high probability to it. The token is technically in the vocabulary but the model treats it as essentially never the right answer. At inference the generation loop keeps sampling other tokens forever, or until it hits the `max_new_tokens` hard cap. The output ends with rambling continuations, made-up follow-up turns, or repetitive padding text. The most common cause of this bug is a chat template change that strips EOS from the training labels, often during a custom preprocessing step.

**EOS is the stop signal**, both at training time and inference time, and the entire generation control flow depends on the model having learned to emit it. During supervised fine-tuning the EOS token sits at the end of every assistant response in the training labels. The cross-entropy loss pushes the model to assign high probability to EOS at that specific position, given everything that came before. By the end of training the model has gradient-aligned EOS to the meaning 'I am done answering.' At inference the generation loop has a straightforward exit condition. After sampling each new token, it checks whether the token equals the model's EOS ID (or any of a configurable list of stop tokens). If yes, stop and return. If no, append and continue. The loop also has a hard cap on `max_new_tokens` to prevent infinite runaway, but in normal operation EOS is what triggers stopping. If EOS never appears in the training labels, the model effectively learns that EOS has zero probability everywhere. The generation loop runs until `max_new_tokens`, at which point it truncates whatever the model was doing. The truncated text typically looks like a complete response followed by random rambling: the model produces a reasonable answer, then continues with hallucinated next-turn user messages, repetitive padding, or wandering tangents because nothing in its training signaled where to stop. The most common root cause is a chat template that strips EOS during preprocessing, often because the template change happened after the fine-tuning recipe was authored and nobody re-verified the rendered output. The fix is to render a sample example end to end and visually confirm the EOS token is present at the close of every assistant span.

Quick reference

TL;DR

EOS is the stop signal. If training labels never include it, the model never learns to emit it and generation runs until max_new_tokens at inference.

Memory aid

Easy to grasp

Imagine teaching a child to read aloud, but you never show them what a period looks like or what it means. They will read each sentence into the next sentence without pausing, because nobody ever taught them where a thought is supposed to stop. The EOS token is the period for a language model. The model learns from examples to put it at the end of a complete response, and the generation system watches for it like a listener watching for a pause. If the training data has the period stripped out, the model fluently produces words forever, because it has no idea that stopping is even an option. The fix is always to put the period back into the training labels.

Key concepts

End-of-sequence token Chat template Loss masking

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: define EOS as the stop signal, walk through the training-time and inference-time mechanics, then enumerate the three failure modes (template strip, tokeniser swap, multi-stop mismatch) and the sanity check that catches each.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face TRL's SFTTrainer warns when training examples lack an EOS token at the end of the assistant turn, exactly because the failure mode is so common.
Llama 4 Maverick fine-tunes use `<|eot_id|>` as the end of turn stop token, which must be both in training labels and in the inference stop_token list.
Qwen 3.5 fine-tunes that mix chat templates often drop `<|im_end|>` from labels, producing post-deployment models that ramble past the assistant turn.
DeepSeek V4 distillation runs explicitly include eos_token in the loss-computed span of every example to avoid the never-stops failure mode.
Production support requests at Hugging Face and Anthropic frequently cite 'model never stops generating' as a top-five fine-tuning bug, traced back to EOS handling almost every time.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf the model emits a stop token but the generation loop does not halt, where is the bug?

QHow does loss masking interact with EOS, and why is this often the silent root cause?

QWhy does a model with no EOS gradient often produce a believable response followed by rambling, rather than rambling from the start?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Stripping EOS during dataset preprocessing because it 'looks like padding' and then wondering why the fine-tuned model rambles forever past the end of its response.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.