Click any words you think contain an error. Click again to unmark.
Mark at least one word to submit.
There are two bugs in this snippet, and both compound to break the same behavior: the model learning when to stop.
The first bug is `add_generation_prompt=True`. That flag is the inference setting. It tells apply_chat_template to render the conversation up to and including the assistant role header (so the model is ready to start generating its reply), but to leave the turn open without the closing EOS token. At inference that is exactly what you want: the model itself will emit EOS when it finishes generating. For training data, you want the opposite: you want the full closed turn including EOS, so the model learns that EOS is the right thing to emit at the end of a response. The fix is `add_generation_prompt=False`.
The second bug is `labels = input_ids[:-1]`. That slice removes the final token of the sequence, which for a properly closed training turn would be EOS. Even if you fix the first bug, this slice undoes the fix by dropping EOS again. The standard recipe is `labels = input_ids.clone()`, then mask the prompt positions to -100 so the loss is computed only on the response. EOS stays as a real, supervised label in the response span.
The consequence of either bug is the same. The model never sees EOS as a target during training, so it learns to put near-zero probability on emitting EOS. At inference, your stop rule fires on EOS, but the model rarely produces one, so generation runs past where the turn should have ended.
**Two compounding bugs** silently break stop behavior in this snippet.
**Bug 1: `add_generation_prompt=True`.** This is the inference setting. It renders the conversation through the assistant role header (e.g. `<|im_start|>assistant\n` for ChatML, `<|start_header_id|>assistant<|end_header_id|>\n\n` for Llama-3) but does NOT append the closing EOS token, because at inference the model is supposed to emit EOS itself. Using this setting for training data means the labels never contain EOS as a target.
The fix is `add_generation_prompt=False`. With that setting apply_chat_template renders the complete turn, including the closing EOS. The training data now has EOS in the right position.
**Bug 2: `labels = input_ids[:-1]`.** This slice strips the final token of the sequence. For a properly closed turn that final token is EOS. Even if Bug 1 is fixed, this slice immediately re-removes the stop signal.
The correct recipe is `labels = input_ids.clone()` followed by masking the prompt positions to -100 so the loss is computed only on the response span. EOS stays as a real, supervised label. The model learns to predict it.
**Consequence.** Either bug alone breaks training. With both, the failure is doubly silent. The model trains to convergence on the response content but never learns to emit EOS, so at inference it never knows when to stop. Output runs past the expected end of turn, hallucinates additional turns (sometimes inventing user messages), and hits max_new_tokens as the only termination condition.
**Verification.** After fixing both bugs, decode a training sample's labels back to a string with skip_special_tokens=False and confirm that the response span ends with EOS (or the model's specific end-of-turn token like `<|im_end|>` or `<|eot_id|>`). If it does not, the recipe is still broken.
Quick reference
TL;DR
Two bugs compound. add_generation_prompt=True is the inference setting (no EOS); labels = input_ids[:-1] then drops the final token anyway. Both must be fixed for the model to learn to stop.
Picture teaching someone when to stop talking by reading them example conversations. Two things can quietly remove the stop signal from your examples. First, you accidentally read aloud only the part up to the moment someone is about to reply, never the reply ending; the listener never hears the closing cue. Second, even when you do read the closing cue, you skip the very last word of every example to save time. After enough examples like that, the listener has no model of when to wrap up. At conversation time they keep going forever. The fix is to read the complete examples, including the final closing cue, every single time.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Rendering data for inference at serving time add_generation_prompt=True is correct here; the model needs the open-turn rendering to know where to start generating.
Pretraining from scratch on raw text without chat structure The chat template does not apply; raw next-token prediction does not need EOS-in-labels handling of this form.
Custom curriculum where partial responses are intentionally trained without EOS Some research recipes deliberately train the model to continue past EOS for specific behaviors; the bug-free recipe is then a different design choice.
Token-level RLHF where rewards apply per token and EOS is handled differently RLHF pipelines often shape the loss around the reward model rather than cross-entropy on EOS as a fixed target.
Quick scan · 2–4 min
Everything important, quickly.
4 min: identify the two errors, explain why each removes EOS from the labels, walk through the correct recipe, then describe the startup assertion that catches the bug class.
Side-by-side
Setting
Train (correct)
Inference (correct)
Buggy train
add_generation_prompt
False
True
True (drops EOS)
labels construction
input_ids.clone() + mask prompt to -100
n/a
input_ids[:-1] (drops final token)
Final position in labels
EOS supervised
n/a
Token before EOS supervised; EOS gone
Inference behavior
Model emits EOS on turn end
Model emits EOS on turn end
Model rarely emits EOS; runs past turn
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face apply_chat_template documentation explicitly notes that add_generation_prompt=True is for inference and warns against using it for training data.
2TRL SFTTrainer handles the chat-template rendering and labels construction correctly out of the box; the bug usually appears in custom training loops.
3Practitioners debugging runaway generation on Llama-3, Qwen, and Mistral fine-tunes regularly trace the bug to one of these two patterns in custom data-prep code.
4Open-source repos like Axolotl and Unsloth ship example chat-template recipes that explicitly avoid both bugs and serve as reference implementations.
5Production fine-tuning vendors include round-trip label verification as a startup assertion precisely because this class of bug is so easy to introduce and so silent at training time.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you write a startup assertion that catches both bugs before training launches?
Sign in or go Pro to see the approach for this follow-up.
QWhy does the bug not show up in training loss curves or eval loss curves?
Sign in or go Pro to see the approach for this follow-up.
QHow does this bug manifest differently across model families?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating add_generation_prompt=True as the training default, or slicing labels with [:-1] to align them with logits. Both quietly remove the EOS from the supervised span and break stop behavior.