Why does dropping the EOS token from SFT labels produce a model that never stops generating?

Flashcard·Easy·4.0 · 0·~30s·Asked atBraintrust Jane Street Mongodb·Relevant atCoreweave Databricks Runway Stability Ai

Attempt it

EOS is the stop signal. The model learns to emit it when a response is complete; the generation loop watches for it and halts. If training labels never include EOS, for instance because a chat template change stripped it before the loss was computed, the model never learns to stop. At inference it runs until max_new_tokens, padding completions with rambling continuations or random follow-up turns.

Your answer

**EOS** stands for end of sequence. It is a single special token in the model's vocabulary that means 'this response is complete.' During training, EOS is the very last target token in every example. The model sees a question and a response, and at the position right after the last word of the response, the target is EOS. The cross-entropy loss pushes the model to assign high probability to EOS at exactly that position, learning that EOS belongs at the end of a complete answer. At inference, generation runs as a loop: sample the next token, append it to the running sequence, feed it back in, repeat. The loop has one natural exit condition: if the sampled token is EOS, stop. The generated text up to but not including EOS is returned as the response. If training labels never contain EOS, the model never learns to assign high probability to it. The token is technically in the vocabulary but the model treats it as essentially never the right answer. At inference the generation loop keeps sampling other tokens forever, or until it hits the `max_new_tokens` hard cap. The output ends with rambling continuations, made-up follow-up turns, or repetitive padding text. The most common cause of this bug is a chat template change that strips EOS from the training labels, often during a custom preprocessing step.

**EOS is the stop signal**, both at training time and inference time, and the entire generation control flow depends on the model having learned to emit it. During supervised fine-tuning the EOS token sits at the end of every assistant response in the training labels. The cross-entropy loss pushes the model to assign high probability to EOS at that specific position, given everything that came before. By the end of training the model has gradient-aligned EOS to the meaning 'I am done answering.' At inference the generation loop has a straightforward exit condition. After sampling each new token, it checks whether the token equals the model's EOS ID (or any of a configurable list of stop tokens). If yes, stop and return. If no, append and continue. The loop also has a hard cap on `max_new_tokens` to prevent infinite runaway, but in normal operation EOS is what triggers stopping. If EOS never appears in the training labels, the model effectively learns that EOS has zero probability everywhere. The generation loop runs until `max_new_tokens`, at which point it truncates whatever the model was doing. The truncated text typically looks like a complete response followed by random rambling: the model produces a reasonable answer, then continues with hallucinated next-turn user messages, repetitive padding, or wandering tangents because nothing in its training signaled where to stop. The most common root cause is a chat template that strips EOS during preprocessing, often because the template change happened after the fine-tuning recipe was authored and nobody re-verified the rendered output. The fix is to render a sample example end to end and visually confirm the EOS token is present at the close of every assistant span.

TL;DR

EOS is the stop signal. If training labels never include it, the model never learns to emit it and generation runs until max_new_tokens at inference.

Memory aid

Easy to grasp

Imagine teaching a child to read aloud, but you never show them what a period looks like or what it means. They will read each sentence into the next sentence without pausing, because nobody ever taught them where a thought is supposed to stop. The EOS token is the period for a language model. The model learns from examples to put it at the end of a complete response, and the generation system watches for it like a listener watching for a pause. If the training data has the period stripped out, the model fluently produces words forever, because it has no idea that stopping is even an option. The fix is always to put the period back into the training labels.

Key concepts

End-of-sequence token Chat template Loss masking

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The EOS token is the smallest piece of a fine-tuned language model and the one most likely to silently destroy its usability in production. The token itself is a single vocabulary entry. Its job is to mean 'the response is complete.' The mechanism by which that meaning gets installed and exploited is precise enough that getting any link in the chain wrong produces a model that never stops generating.

The symptom is unmistakable. A user asks a question, the model produces a believable answer, and then keeps going. Sometimes it produces a hallucinated next user turn. Sometimes it loops on repetitive padding. Sometimes it wanders off into an unrelated tangent. The output stops only when the inference loop hits its max_new_tokens hard cap, often producing several thousand tokens of unwanted text per request.

This deep dive walks through what EOS actually does during training, how the generation loop uses it at inference, the three specific failure modes that produce the never-stops symptom, and the operational hygiene that prevents the bug from ever shipping.

What EOS does during training

Supervised fine-tuning trains the model on a sequence of next-token prediction tasks. For each position in the training sequence, the model produces a probability distribution over the vocabulary, and the cross-entropy loss compares that distribution to the ground-truth next token.

The EOS token sits at the end of every assistant response in the training labels. After the last real content token, the next target is EOS. The loss at that position pushes the model to assign high probability to EOS given everything that came before, which means the model learns to recognise 'this response is now complete' as a context that calls for EOS.

The loss masking detail matters. In chat-style SFT, the typical mask sets the loss to ignore-index on all system and user tokens, contributing gradient only on assistant tokens. The EOS that ends an assistant turn must be inside the assistant-mask span; if the mask boundary places EOS outside (treating it as part of the next user turn or as padding), no gradient flows and the model never learns to emit it.

A correctly trained chat model will, by the end of training, place high probability on EOS at exactly the positions where assistant responses end and lower probability elsewhere. That learned distribution is what makes generation stop at the right time.

How the generation loop uses EOS

The three failure modes and their signatures

Hygiene that prevents the bug from shipping

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Hugging Face TRL's SFTTrainer warns when training examples lack an EOS token at the end of the assistant turn, exactly because the failure mode is so common.
Llama 4 Maverick fine-tunes use `<|eot_id|>` as the end of turn stop token, which must be both in training labels and in the inference stop_token list.

What an interviewer would ask next. Try answering before peeking at the approach.

QIf the model emits a stop token but the generation loop does not halt, where is the bug?

Trace the inference-side configuration. The loop's stop list (eos_token_id, stop_strings, eos_token_id_list) must include the exact token ID the model emits; mismatched lists are a common cause of trained models that look broken at serve time.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Stripping EOS during dataset preprocessing because it 'looks like padding' and then wondering why the fine-tuned model rambles forever past the end of its response.

60 second bullets to scan on the way to the call.

The mechanical role of EOS in the training cross-entropy loss
How the inference generation loop uses EOS to exit

Primary sources. Browse if you want the original framing.