Spot the bug: a Llama-3 fine-tune that skips the end-of-turn token

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Quick reference

TL;DR

The template drops the `<|eot_id|>` end-of-turn token, so the model never learns to stop and drifts into invented turns at inference.

Memory aid

Easy to grasp

Imagine teaching someone to write letters, but every example you give them runs all the letters together with no full stops or sign-offs. They learn the words fine, yet they never learn where one letter ends. So when they write their own, they just keep going, signing your name then starting a fresh letter from a stranger. The fix is to put the sign-off back in every example so they learn that a turn has an ending. Here the sign-off is a special end-of-turn marker the model is supposed to emit. Skip it during training and the model has no idea it should stop talking, so at answer time it rambles past its own reply into a fake follow-up.

Key concepts

Chat template formatting Special tokens Loss masking Supervised fine-tuning

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: locate the dropped end-of-turn token, explain the train-time and inference-time consequences, cover the loss-masking boundary, then give the apply_chat_template fix plus a round-trip verification.

Code example

python

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
msgs = [
    {"role": "user", "content": user_msg},
    {"role": "assistant", "content": assistant_msg},
]
# Builds the exact Llama-3 string, inserting <|eot_id|> after every turn.
text = tok.apply_chat_template(msgs, tokenize=False)
assert "<|eot_id|>" in text  # sanity-check the terminator is present

Side-by-side

Aspect	Buggy hand-built template	apply_chat_template()
End-of-turn token	Dropped between turns and after assistant	Inserted after every turn automatically
Stop behaviour learned	No, the model never sees the terminator	Yes, the terminator is in every example
Cross-model portability	Breaks when the base model changes	Adapts to each model's own template
Failure visibility	Silent, training loss looks fine	Round-trip decode confirms correctness

Real-world examples

Real products, models, and research that use this idea.

Hugging Face TRL's SFTTrainer ships a DataCollatorForCompletionOnlyLM that masks prompt tokens, and its docs warn that the response template must match the model's own chat template.
Meta's Llama 3 model card and tokenizer config define `<|eot_id|>` as the turn terminator, and `apply_chat_template()` in transformers inserts it automatically.
Unsloth and Axolotl both document train-on-responses-only masking and ship per-model templates precisely so teams stop hand-rolling Llama-3, Qwen, and Mistral formats.
Modern frontier chat models like Claude Opus 4.7 and Llama 4 rely on strict special-token grammars where a single misplaced turn marker degrades multi-turn behaviour.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does training loss stay healthy while the model silently forgets to stop generating?

QHow would you confirm whether the stop token is inside the loss mask versus the template?

QIf you must hand-build templates for a custom serving stack, how do you keep them safe?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Building the prompt string by hand instead of calling the tokenizer's chat template. A single dropped end of turn token silently teaches the model never to stop.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.