Spot the bug: a Llama-3 fine-tune that skips the end of turn token
Click any words you think contain an error. Click again to unmark.
The template drops the `<|eot_id|>` end-of-turn token, so the model never learns to stop and drifts into invented turns at inference.
Imagine teaching someone to write letters, but every example you give them runs all the letters together with no full stops or sign-offs. They learn the words fine, yet they never learn where one letter ends. So when they write their own, they just keep going, signing your name then starting a fresh letter from a stranger. The fix is to put the sign-off back in every example so they learn that a turn has an ending. Here the sign-off is a special end-of-turn marker the model is supposed to emit. Skip it during training and the model has no idea it should stop talking, so at answer time it rambles past its own reply into a fake follow-up.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Chat-template bugs are the quiet assassins of fine-tuning. The code runs, the schema validates, the loss curve descends smoothly, and the eval harness reports a plausible number. Then you ship, and the model rambles past its own answer into a hallucinated next turn. Nothing in the training logs warned you, because the defect lives in the exact place logs do not look: the byte-level layout of special tokens around each turn boundary.
This question puts that failure under a microscope. A team is fine-tuning Llama-3-8B on support conversations and hand-assembling the tokenised string. They get the role headers right, they get the loss masking conceptually right, but they drop one token: the end-of-turn marker <|eot_id|> that Llama-3 places after every turn. That single omission is enough to break the model's ability to stop.
To answer well you need three things. First, recognise the specific missing token and where it belongs. Second, explain the mechanism, why a missing terminator at training time produces non-stopping behaviour at inference time. Third, give the durable fix, which is procedural rather than a one-line patch: stop building these strings by hand and let the model's own tokenizer do it.
What the Llama-3 chat template actually requires
Llama-3 uses a small grammar of special tokens to structure a conversation. A turn opens with <|start_header_id|>, the role name, then <|end_header_id|> and two newlines. The content follows. Critically, the turn then closes with <|eot_id|>, the end-of-turn token, before the next header begins. There is also a single <|begin_of_text|> token at the very start of the sequence, which the snippet does get right.
A correctly formatted two-turn example therefore looks like a user header, the user content, <|eot_id|>, then an assistant header, the assistant content, and a final <|eot_id|>. The terminator is not decoration. It is the boundary symbol the whole format hangs on, and it is a single registered vocabulary entry, not three separate characters that the tokeniser would otherwise split.
The buggy snippet writes the user content and jumps straight to the next <|start_header_id|> with no <|eot_id|> in between, and it ends the assistant content without one either. The string is now off-grammar. It is not the format the base instruct model was aligned on, and it no longer carries the symbol that means a turn has ended.
This distinction between matching the template and improvising one is the heart of the question. The base model already learned, during its own instruction tuning, that an assistant turn ends at <|eot_id|>. By stripping that token from your fine-tuning data you actively teach against the prior the base model arrived with. You are not just omitting information; you are overwriting a correct behaviour with an incorrect one, which is why the damage is larger than a missing newline would suggest.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
msgs = [
{"role": "user", "content": user_msg},
{"role": "assistant", "content": assistant_msg},
]
# Builds the exact Llama-3 string, inserting <|eot_id|> after every turn.
text = tok.apply_chat_template(msgs, tokenize=False)
assert "<|eot_id|>" in text # sanity-check the terminator is present| Aspect | Buggy hand-built template | apply_chat_template() |
|---|---|---|
| End-of-turn token | Dropped between turns and after assistant | Inserted after every turn automatically |
| Stop behaviour learned | No, the model never sees the terminator | Yes, the terminator is in every example |
| Cross-model portability | Breaks when the base model changes | Adapts to each model's own template |
| Failure visibility | Silent, training loss looks fine | Round-trip decode confirms correctness |
Real products, models, and research that use this idea.
- Hugging Face TRL's SFTTrainer ships a DataCollatorForCompletionOnlyLM that masks prompt tokens, and its docs warn that the response template must match the model's own chat template.
- Meta's Llama 3 model card and tokenizer config define `<|eot_id|>` as the turn terminator, and `apply_chat_template()` in transformers inserts it automatically.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does training loss stay healthy while the model silently forgets to stop generating?
Loss is averaged over supervised tokens. A missing terminator removes one token's signal per example, barely moving the average while erasing the entire stop behaviour the model needed to learn.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Building the prompt string by hand instead of calling the tokenizer's chat template. A single dropped end of turn token silently teaches the model never to stop.
60 second bullets to scan on the way to the call.
Which special token terminates a turn in the Llama-3 chat template
Why a missing terminator breaks inference even when training loss looks fine
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.