Spot the bug: chat-template code that drops the EOS from the labels.

Spot the error·Medium·4.0 · 0·~2 min·Relevant atDatabricks Meta OpenAI

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

There are two bugs in this snippet, and both compound to break the same behavior: the model learning when to stop. The first bug is `add_generation_prompt=True`. That flag is the inference setting. It tells apply_chat_template to render the conversation up to and including the assistant role header (so the model is ready to start generating its reply), but to leave the turn open without the closing EOS token. At inference that is exactly what you want: the model itself will emit EOS when it finishes generating. For training data, you want the opposite: you want the full closed turn including EOS, so the model learns that EOS is the right thing to emit at the end of a response. The fix is `add_generation_prompt=False`. The second bug is `labels = input_ids[:-1]`. That slice removes the final token of the sequence, which for a properly closed training turn would be EOS. Even if you fix the first bug, this slice undoes the fix by dropping EOS again. The standard recipe is `labels = input_ids.clone()`, then mask the prompt positions to -100 so the loss is computed only on the response. EOS stays as a real, supervised label in the response span. The consequence of either bug is the same. The model never sees EOS as a target during training, so it learns to put near-zero probability on emitting EOS. At inference, your stop rule fires on EOS, but the model rarely produces one, so generation runs past where the turn should have ended.

**Two compounding bugs** silently break stop behavior in this snippet. **Bug 1: `add_generation_prompt=True`.** This is the inference setting. It renders the conversation through the assistant role header (e.g. `<|im_start|>assistant\n` for ChatML, `<|start_header_id|>assistant<|end_header_id|>\n\n` for Llama-3) but does NOT append the closing EOS token, because at inference the model is supposed to emit EOS itself. Using this setting for training data means the labels never contain EOS as a target. The fix is `add_generation_prompt=False`. With that setting apply_chat_template renders the complete turn, including the closing EOS. The training data now has EOS in the right position. **Bug 2: `labels = input_ids[:-1]`.** This slice strips the final token of the sequence. For a properly closed turn that final token is EOS. Even if Bug 1 is fixed, this slice immediately re-removes the stop signal. The correct recipe is `labels = input_ids.clone()` followed by masking the prompt positions to -100 so the loss is computed only on the response span. EOS stays as a real, supervised label. The model learns to predict it. **Consequence.** Either bug alone breaks training. With both, the failure is doubly silent. The model trains to convergence on the response content but never learns to emit EOS, so at inference it never knows when to stop. Output runs past the expected end of turn, hallucinates additional turns (sometimes inventing user messages), and hits max_new_tokens as the only termination condition. **Verification.** After fixing both bugs, decode a training sample's labels back to a string with skip_special_tokens=False and confirm that the response span ends with EOS (or the model's specific end-of-turn token like `<|im_end|>` or `<|eot_id|>`). If it does not, the recipe is still broken.

Quick reference

TL;DR

Two bugs compound. add_generation_prompt=True is the inference setting (no EOS); labels = input_ids[:-1] then drops the final token anyway. Both must be fixed for the model to learn to stop.

Memory aid

Easy to grasp

Picture teaching someone when to stop talking by reading them example conversations. Two things can quietly remove the stop signal from your examples. First, you accidentally read aloud only the part up to the moment someone is about to reply, never the reply ending; the listener never hears the closing cue. Second, even when you do read the closing cue, you skip the very last word of every example to save time. After enough examples like that, the listener has no model of when to wrap up. At conversation time they keep going forever. The fix is to read the complete examples, including the final closing cue, every single time.

Key concepts

Chat templates EOS token Loss masking Stop tokens

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: identify the two errors, explain why each removes EOS from the labels, walk through the correct recipe, then describe the startup assertion that catches the bug class.

Side-by-side

Setting	Train (correct)	Inference (correct)	Buggy train
add_generation_prompt	False	True	True (drops EOS)
labels construction	input_ids.clone() + mask prompt to -100	n/a	input_ids[:-1] (drops final token)
Final position in labels	EOS supervised	n/a	Token before EOS supervised; EOS gone
Inference behavior	Model emits EOS on turn end	Model emits EOS on turn end	Model rarely emits EOS; runs past turn

Real-world examples

Real products, models, and research that use this idea.

Hugging Face apply_chat_template documentation explicitly notes that add_generation_prompt=True is for inference and warns against using it for training data.
TRL SFTTrainer handles the chat-template rendering and labels construction correctly out of the box; the bug usually appears in custom training loops.
Practitioners debugging runaway generation on Llama-3, Qwen, and Mistral fine-tunes regularly trace the bug to one of these two patterns in custom data-prep code.
Open-source repos like Axolotl and Unsloth ship example chat-template recipes that explicitly avoid both bugs and serve as reference implementations.
Production fine-tuning vendors include round-trip label verification as a startup assertion precisely because this class of bug is so easy to introduce and so silent at training time.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you write a startup assertion that catches both bugs before training launches?

QWhy does the bug not show up in training loss curves or eval loss curves?

QHow does this bug manifest differently across model families?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating add_generation_prompt=True as the training default, or slicing labels with [:-1] to align them with logits. Both quietly remove the EOS from the supervised span and break stop behavior.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.