After editing the Jinja chat template, your fine-tuned Llama 3 prints raw special token text. What broke?

Spot the error·Medium·4.0 · 0·~2 min·Asked atHugging Face Ironclad Kpmg·Relevant atMeta

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Two causes work together to produce the bug. **Cause 1: special-token handling in the template render.** The string `<|start_header_id|>` in the Jinja template is supposed to become a SINGLE special-token id when the rendered template gets tokenized. That conversion only happens if the tokenizer has those strings registered as special tokens AND the encoding call uses `add_special_tokens=True` (or a method that respects the special-token map). If the rendered template is tokenized with `add_special_tokens=False` or with a tokenizer that does not have the strings registered, `<|start_header_id|>` tokenizes as multiple ordinary text tokens (something like `'<|'`, `'start_header_id'`, `'|>'`). The model now sees text tokens in positions where the original training expected special-token ids. **Cause 2: training/inference template drift.** The fine-tuning data was generated with the ORIGINAL chat template. The model learned to predict whatever those original tokens were in whatever positions they appeared. Then the MODIFIED template was deployed for inference, with a custom system-prefix prepended. The model now sees prompts in a different format than it learned, and its predictions reflect the training-time distribution, not the inference-time one. Combined: the model has learned to predict literal `<|start_header_id|>` text in certain positions (because the training data had it as text, or because the template change shifted what comes before the assistant turn), and outputs that literal at inference. **Fix.** Render the template. Tokenize the rendered string with the special-token map active. Decode the tokenized ids back. Diff against the fine-tuning data byte by byte. If they differ, fix the template or the special-token handling, then re-tune the model. The template and the data must match exactly.

Two coupled bugs. **Bug 1: special-token map.** `<|start_header_id|>` becomes a single special-token id only if (1) the tokenizer has it registered in special_tokens_map, AND (2) the encoding uses `add_special_tokens=True` (or `apply_chat_template` which uses the map). If either fails, the string tokenizes as multiple text tokens, and the model learns to predict those text tokens. Common variant: Jinja template renders to a string, the string is saved as fine-tune data, the loader tokenizes with `add_special_tokens=False`. Every `<|start_header_id|>` in training is ordinary text. The model learns to emit it as text. At inference the literal occasionally appears. **Bug 2: training/inference template drift.** Fine-tune used the ORIGINAL template. The MODIFIED template was deployed. Adding `SYSTEM-PREFIX:` shifts the position of `<|start_header_id|>` relative to other tokens, which the model never saw at training. **Detection.** Render the template on a sample messages list. Tokenize with `add_special_tokens=True`. Decode the ids. Do the same with the training-time template. Diff. Any difference means inconsistent behavior. Llama 3 special-token ids: `<|begin_of_text|>=128000`, `<|start_header_id|>=128006`, `<|end_header_id|>=128007`, `<|eot_id|>=128009`. If you see these single ids, handling is correct; if you see multi-token sequences for special-token strings, the encoding flags are wrong. **Fix.** Revert template to training-time version, OR regenerate training data with the new template using correct special-token tokenization and re-tune. Template and training data must match byte by byte.

TL;DR

Two bugs: template strings like '<|start_header_id|>' can tokenize as text without the special-token map, AND fine-tune data used a different template than deployed. Decode + diff against training.

Memory aid

Easy to grasp

Imagine you teach a child a code language for marking sections in their homework: 'BEGIN-MATH' means a math problem starts. Now you change the rule to 'BEGINMATH' (one word) without telling the child. The child keeps writing 'BEGIN-MATH' from old habit, because that is what they learned. The same happens with chat templates and fine-tuning. The model was trained on the original template. You modified the template later for inference. Some of the role-marker strings tokenize differently now, or they were stored as text instead of as the single special-id they should have been. The model occasionally outputs the raw text it learned, because that is what its training data showed. The fix is to keep the training-time and inference-time formats identical, byte by byte.

Key concepts

Jinja template Fine-tuning Special tokens

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

This bug is one of the most subtle in the fine-tune and deploy pipeline. The model trained correctly. The deploy succeeded. Inference returns plausible text. Then, occasionally, a response includes a raw <|start_header_id|> or similar special-token string in the output. The team blames sampling, blames temperature, blames the prompt, eventually arrives at the template, and only then realizes that template modifications interact with the special-token map and the training data in non-obvious ways.

The correct diagnosis names two coupled bugs. First, special-token handling: the string <|start_header_id|> becomes a single special-token id only when the tokenizer's special-token map is active during encoding. Without it, the string fragments into multiple text tokens, which the model can learn to emit as content. Second, training/inference template drift: modifying chat_template after fine-tuning means the model learned from one format and gets prompted in another. Both bugs compound; the model has learned to predict the literal special-token text in positions where the modified template did not match its training.

The rest of this explanation walks each bug in detail, names the decode and diff diagnostic that surfaces them, explains why both have to be fixed together, and closes on the production discipline of treating templates as part of the trained model artifact.

Bug 1: special-token handling and the string versus id distinction

The critical mechanism: the tokenizer's encode method takes a flag that controls whether special-token strings in the input map to their dedicated ids or get tokenized as ordinary text. With add_special_tokens=True (or specific APIs like apply_chat_template that respect the special-token map), the string <|start_header_id|> in the input becomes the single id 128006. With add_special_tokens=False, the string tokenizes as multiple ordinary BPE tokens (something like <, |, start, _header_id, |, >).

This matters at training data preparation time. If the fine-tuning data preparation pipeline:

Calls the Jinja template to render messages into a string.
Writes the string to disk (parquet, jsonl).
Loads the string and tokenizes with add_special_tokens=False.

Then every <|start_header_id|> in the training data is stored as multiple ordinary text tokens. The model learns to emit those text tokens at the positions they appeared. At inference, the same text tokens occasionally come out as content, which is the symptom.

The fix at this layer: use apply_chat_template(..., tokenize=True) directly so the rendering and the special-token aware tokenization happen in one step. Or, if the pipeline must render then tokenize separately, ensure the tokenize step uses add_special_tokens=True and the tokenizer has the special tokens registered.

Bug 2: training/inference template drift

Why both bugs typically appear together and how they compound

Detection: the decode and diff diagnostic

Production discipline: template as part of the model artifact

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

HuggingFace tokenizers expose added_tokens_encoder / added_tokens_decoder which control whether a registered string maps to a single id; apply_chat_template uses this map automatically.
Llama 3's tokenizer registers <|begin_of_text|>, <|start_header_id|>, <|end_header_id|>, <|eot_id|> as special tokens with ids 128000, 128006, 128007, 128009 respectively.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you write a CI test that catches this class of bug before deploying a fine-tune?

On training-data prep, assert that every rendered example contains the expected special-token ids (e.g., 128006 for <|start_header_id|> on Llama 3) at the expected positions. On deployment, render the inference template for a fixture messages list, tokenize, and assert the resulting id sequence matches a recorded golden from training. Any divergence is a deploy blocker. This catches both bugs at once.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Modifying the chat template after fine-tuning, so the training-time and inference-time formats diverge; or rendering templates to text without the special-token map active, so the role-marker strings tokenize as ordinary text.

60 second bullets to scan on the way to the call.

How a string like <|start_header_id|> becomes a single special-token id vs multiple text tokens.
Why add_special_tokens flag and the special-token map control the conversion.

Primary sources. Browse if you want the original framing.