A hand-built chat prompt performs worse than apply_chat_template. Spot the tokenization issues.

Spot the error·Medium·4.0 · 0·~2 min·Relevant atAnthropic OpenAI

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

There are two tokenization-level errors, and they compound. **Error 1: bare role labels are not the model's special role tokens.** The model was instruction-tuned on a specific format with special tokens that mark role boundaries. For Llama 3, that is `<|start_header_id|>system<|end_header_id|>`, then content, then `<|eot_id|>`. For OpenAI-tuned models, it is `<|im_start|>system` and `<|im_end|>`. The hand-rolled string `system:` tokenizes as ordinary text tokens (something like `'system'` plus `':'`), which the model has no learned signal to interpret as a turn boundary. **Error 2: the trailing `assistant:` creates a leading-space boundary issue.** In BPE, the token ` Hello` (with leading space) and `Hello` (without) are different ids. At training time, the assistant turn often starts immediately after a header token with no extra space; hand-rolled `assistant:` followed by generation may make the first generated word land on its leading-space variant instead of the no-space variant the model expects. The two errors compound: missing special tokens means weak turn-boundary signal, and shifted first-token id means the first word is off-distribution. The model still produces output (it always does), but instruction-following degrades subtly. **Fix.** Always use `tokenizer.apply_chat_template(messages, add_generation_prompt=True)`. This applies the model's authoritative Jinja chat template, inserts the right special tokens, and gets the leading-space behavior correct per model. Never hand-roll role formatting for an instruction-tuned model.

**Two coupled errors at the tokenization layer.** **Error 1: bare role labels do not produce special role tokens.** Each instruction-tuned model has a specific format it was trained on with special tokens. Llama 3 uses `<|start_header_id|>{role}<|end_header_id|>` and `<|eot_id|>`. ChatML-style models use `<|im_start|>{role}` and `<|im_end|>`. The hand-rolled string `system:` tokenizes as ordinary text tokens, not as those special tokens. The model has no learned signal that these strings mark conversation structure, so turn parsing degrades in ways that look like instruction-following failures but are actually format-recognition failures. **Error 2: leading-space boundary on the first generated token.** The trailing `assistant:` followed by model generation creates a subtle id mismatch. In byte-level BPE, ` Hello` and `Hello` are different ids. At training time, the assistant turn often begins immediately after the role header with no leading space (the header token provides the separator). Hand-rolled `assistant:` followed by generation may put the first generated word on its leading-space variant or its no-space variant in a different way than training, shifting the first-token id distribution off-distribution. **Why both errors are silent.** No exception fires. The model still generates fluent text. Quality degrades subtly in ways that show up as 'the model is not following instructions well' rather than as a clear bug. Developers blame the prompt, the temperature, the system message, anything except the tokenization layer. **Fix.** `tokenizer.apply_chat_template(messages, add_generation_prompt=True)`. This applies the authoritative Jinja chat template from `tokenizer_config.json`, inserts the right special tokens, places generations starts correctly, and is portable across models (the template ships per model). Never hand-roll role formatting for an instruction-tuned model.

Quick reference

TL;DR

Two errors. Bare 'system:'/'user:'/'assistant:' tokenize as text, not as the special role tokens the model was trained on. Trailing 'assistant:' also shifts the first-token id via leading-space. Use apply_chat_template.

Memory aid

Easy to grasp

Imagine writing a play. The script you give to the actors uses real, agreed-upon labels: ACT 1 SCENE 2, NARRATOR, JANE. They were trained to look for those exact labels. Now you hand them a script that uses 'act-one scene-two', 'narrator-says', 'jane:' instead. The actors can sort of figure it out, but they hesitate at every transition because none of the labels match what they learned to expect. The same thing happens with hand-rolled chat templates. The model was trained on special role tokens like <|start_header_id|>system<|end_header_id|>. Plain 'system:' is not those special tokens; it is just text the model has to guess about. Plus, the spacing where the model is supposed to start speaking shifts the first word's token id, which is another small thing the model has to compensate for.

Key concepts

apply_chat_templateChat template Leading-space boundary

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Open with the framing that hand-rolling chat formatting is a category error: the template is part of the trained model artifact. Walk the two coupled bugs: bare role labels are not the special role tokens the model learned to attend to, and the trailing 'assistant:' shifts the first generated token id via the leading-space boundary. Cover the diagnosis (decode and diff against apply_chat_template, logit-distribution comparison). Close on the fix: apply_chat_template is non-negotiable for production, per-model template loading prevents cross-family bugs.

Code example

python

# Wrong: hand-rolled bare-label template
def build_prompt(sys, msg):
    return f'system: {sys}\nuser: {msg}\nassistant:'
prompt = build_prompt(system_text, user_msg)
ids = tokenizer.encode(prompt, add_special_tokens=False)
# Bug 1: 'system:' etc are ordinary text, not special role tokens
# Bug 2: trailing 'assistant:' shifts first-token id distribution

# Right: apply_chat_template
messages = [
    {'role': 'system', 'content': system_text},
    {'role': 'user',   'content': user_msg},
]
ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors='pt',
)
# Diagnostic: decode and compare
print(tokenizer.decode(ids[0]))
# Shows the actual special tokens the model expects

Real-world examples

Real products, models, and research that use this idea.

HuggingFace transformers ships apply_chat_template that loads the Jinja template from tokenizer_config.json for Llama 3+, Mistral, Qwen 3.5, and other instruction-tuned models.
Llama 3's documented chat format uses <|begin_of_text|> + <|start_header_id|>{role}<|end_header_id|> + content + <|eot_id|>, none of which appear in a hand-rolled 'system:/user:/assistant:' template.
OpenAI's chat completions API hides this entirely by accepting structured role messages, so the server-side rendering applies the right template per model.
Anthropic's messages API for Claude similarly takes typed role turns, eliminating the hand-roll category of error.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf you add the right special tokens manually instead of using apply_chat_template, what could still go wrong?

QYour team supports five different open-weight models. How do you avoid a per-model template bug?

QHow would you detect this class of bug in an existing codebase before users notice?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating hand-rolled chat formatting as equivalent to the model's trained chat format, missing both the special role-token mismatch and the leading-space boundary issue.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.