Click any words you think contain an error. Click again to unmark.
Mark at least one word to submit.
Two causes work together to produce the bug.
**Cause 1: special-token handling in the template render.** The string `<|start_header_id|>` in the Jinja template is supposed to become a SINGLE special-token id when the rendered template gets tokenized. That conversion only happens if the tokenizer has those strings registered as special tokens AND the encoding call uses `add_special_tokens=True` (or a method that respects the special-token map). If the rendered template is tokenized with `add_special_tokens=False` or with a tokenizer that does not have the strings registered, `<|start_header_id|>` tokenizes as multiple ordinary text tokens (something like `'<|'`, `'start_header_id'`, `'|>'`). The model now sees text tokens in positions where the original training expected special-token ids.
**Cause 2: training/inference template drift.** The fine-tuning data was generated with the ORIGINAL chat template. The model learned to predict whatever those original tokens were in whatever positions they appeared. Then the MODIFIED template was deployed for inference, with a custom system-prefix prepended. The model now sees prompts in a different format than it learned, and its predictions reflect the training-time distribution, not the inference-time one.
Combined: the model has learned to predict literal `<|start_header_id|>` text in certain positions (because the training data had it as text, or because the template change shifted what comes before the assistant turn), and outputs that literal at inference.
**Fix.** Render the template. Tokenize the rendered string with the special-token map active. Decode the tokenized ids back. Diff against the fine-tuning data byte by byte. If they differ, fix the template or the special-token handling, then re-tune the model. The template and the data must match exactly.
Two coupled bugs.
**Bug 1: special-token map.** `<|start_header_id|>` becomes a single special-token id only if (1) the tokenizer has it registered in special_tokens_map, AND (2) the encoding uses `add_special_tokens=True` (or `apply_chat_template` which uses the map). If either fails, the string tokenizes as multiple text tokens, and the model learns to predict those text tokens.
Common variant: Jinja template renders to a string, the string is saved as fine-tune data, the loader tokenizes with `add_special_tokens=False`. Every `<|start_header_id|>` in training is ordinary text. The model learns to emit it as text. At inference the literal occasionally appears.
**Bug 2: training/inference template drift.** Fine-tune used the ORIGINAL template. The MODIFIED template was deployed. Adding `SYSTEM-PREFIX:` shifts the position of `<|start_header_id|>` relative to other tokens, which the model never saw at training.
**Detection.** Render the template on a sample messages list. Tokenize with `add_special_tokens=True`. Decode the ids. Do the same with the training-time template. Diff. Any difference means inconsistent behavior. Llama 3 special-token ids: `<|begin_of_text|>=128000`, `<|start_header_id|>=128006`, `<|end_header_id|>=128007`, `<|eot_id|>=128009`. If you see these single ids, handling is correct; if you see multi-token sequences for special-token strings, the encoding flags are wrong.
**Fix.** Revert template to training-time version, OR regenerate training data with the new template using correct special-token tokenization and re-tune. Template and training data must match byte by byte.
Quick reference
TL;DR
Two bugs: template strings like '<|start_header_id|>' can tokenize as text without the special-token map, AND fine-tune data used a different template than deployed. Decode + diff against training.
Imagine you teach a child a code language for marking sections in their homework: 'BEGIN-MATH' means a math problem starts. Now you change the rule to 'BEGINMATH' (one word) without telling the child. The child keeps writing 'BEGIN-MATH' from old habit, because that is what they learned. The same happens with chat templates and fine-tuning. The model was trained on the original template. You modified the template later for inference. Some of the role-marker strings tokenize differently now, or they were stored as text instead of as the single special-id they should have been. The model occasionally outputs the raw text it learned, because that is what its training data showed. The fix is to keep the training-time and inference-time formats identical, byte by byte.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Using a base (base (not instruction-tuned)) model where there is no chat_template at all. Base models do not have a trained chat format; modifying or maintaining one is the wrong concern. Format input as plain text.
Calling a hosted API like OpenAI or Anthropic that handles role formatting server-side. The API applies the template internally; the client cannot modify it and these bugs do not apply at the API boundary.
Pure inference of an off the shelf instruct model with no fine-tuning. If you have not modified the template, the model's training-time and inference-time formats match by default; the bug only appears when you fine-tune or modify the template.
Quick scan · 2–4 min
Everything important, quickly.
Open with the framing that template and training data are coupled. Walk the two bugs: special-token handling (the string vs id distinction, controlled by add_special_tokens and the special-token map) and training/inference template drift (modify template after fine-tune means re-tune). Cover the diagnostic: render template, tokenize, decode, diff against training data. Close on the production discipline of CI tests that assert special-token ids appear at expected positions in training data and golden inference fixtures.
Real-world examples
Real products, models, and research that use this idea.
1HuggingFace tokenizers expose added_tokens_encoder / added_tokens_decoder which control whether a registered string maps to a single id; apply_chat_template uses this map automatically.
2Llama 3's tokenizer registers <|begin_of_text|>, <|start_header_id|>, <|end_header_id|>, <|eot_id|> as special tokens with ids 128000, 128006, 128007, 128009 respectively.
3Common fine-tuning frameworks (axolotl, LLaMA-Factory, unsloth) document that modifying chat_template after fine-tuning requires re-tuning, with explicit warnings about the data-template drift.
4vLLM and Text Generation Inference both warn at load time when the tokenizer's special-token map and the chat_template do not match expected patterns for a given model family.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you write a CI test that catches this class of bug before deploying a fine-tune?
Sign in or go Pro to see the approach for this follow-up.
QIf you really need to modify the chat_template without re-tuning, what is the most you can change safely?
Sign in or go Pro to see the approach for this follow-up.
QYour fine-tune was done correctly but at inference time you're seeing intermittent raw special-token output anyway. What other causes are possible?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Modifying the chat template after fine-tuning, so the training-time and inference-time formats diverge; or rendering templates to text without the special-token map active, so the role-marker strings tokenize as ordinary text.