A hand built chat prompt performs worse than apply_chat_template. Spot the tokenization issues.

Spot the error·Medium·4.0 · 0·~2 min·Asked atAmd Anthropic Neptune Ai·Relevant atOpenAI

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

There are two tokenization-level errors, and they compound. **Error 1: bare role labels are not the model's special role tokens.** The model was instruction-tuned on a specific format with special tokens that mark role boundaries. For Llama 3, that is `<|start_header_id|>system<|end_header_id|>`, then content, then `<|eot_id|>`. For OpenAI-tuned models, it is `<|im_start|>system` and `<|im_end|>`. The hand-rolled string `system:` tokenizes as ordinary text tokens (something like `'system'` plus `':'`), which the model has no learned signal to interpret as a turn boundary. **Error 2: the trailing `assistant:` creates a leading-space boundary issue.** In BPE, the token ` Hello` (with leading space) and `Hello` (without) are different ids. At training time, the assistant turn often starts immediately after a header token with no extra space; hand-rolled `assistant:` followed by generation may make the first generated word land on its leading-space variant instead of the no-space variant the model expects. The two errors compound: missing special tokens means weak turn-boundary signal, and shifted first-token id means the first word is off-distribution. The model still produces output (it always does), but instruction-following degrades subtly. **Fix.** Always use `tokenizer.apply_chat_template(messages, add_generation_prompt=True)`. This applies the model's authoritative Jinja chat template, inserts the right special tokens, and gets the leading-space behavior correct per model. Never hand-roll role formatting for an instruction-tuned model.

**Two coupled errors at the tokenization layer.** **Error 1: bare role labels do not produce special role tokens.** Each instruction-tuned model has a specific format it was trained on with special tokens. Llama 3 uses `<|start_header_id|>{role}<|end_header_id|>` and `<|eot_id|>`. ChatML-style models use `<|im_start|>{role}` and `<|im_end|>`. The hand-rolled string `system:` tokenizes as ordinary text tokens, not as those special tokens. The model has no learned signal that these strings mark conversation structure, so turn parsing degrades in ways that look like instruction-following failures but are actually format-recognition failures. **Error 2: leading-space boundary on the first generated token.** The trailing `assistant:` followed by model generation creates a subtle id mismatch. In byte-level BPE, ` Hello` and `Hello` are different ids. At training time, the assistant turn often begins immediately after the role header with no leading space (the header token provides the separator). Hand-rolled `assistant:` followed by generation may put the first generated word on its leading-space variant or its no-space variant in a different way than training, shifting the first-token id distribution off-distribution. **Why both errors are silent.** No exception fires. The model still generates fluent text. Quality degrades subtly in ways that show up as 'the model is not following instructions well' rather than as a clear bug. Developers blame the prompt, the temperature, the system message, anything except the tokenization layer. **Fix.** `tokenizer.apply_chat_template(messages, add_generation_prompt=True)`. This applies the authoritative Jinja chat template from `tokenizer_config.json`, inserts the right special tokens, places generations starts correctly, and is portable across models (the template ships per model). Never hand-roll role formatting for an instruction-tuned model.

TL;DR

Two errors. Bare 'system:'/'user:'/'assistant:' tokenize as text, not as the special role tokens the model was trained on. Trailing 'assistant:' also shifts the first-token id via leading-space. Use apply_chat_template.

Memory aid

Easy to grasp

Imagine writing a play. The script you give to the actors uses real, agreed-upon labels: ACT 1 SCENE 2, NARRATOR, JANE. They were trained to look for those exact labels. Now you hand them a script that uses 'act-one scene-two', 'narrator-says', 'jane:' instead. The actors can sort of figure it out, but they hesitate at every transition because none of the labels match what they learned to expect. The same thing happens with hand-rolled chat templates. The model was trained on special role tokens like <|start_header_id|>system<|end_header_id|>. Plain 'system:' is not those special tokens; it is just text the model has to guess about. Plus, the spacing where the model is supposed to start speaking shifts the first word's token id, which is another small thing the model has to compensate for.

Key concepts

apply_chat_templateChat template Leading-space boundary

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

This snippet ships in production with depressing regularity. It looks reasonable, runs without error, and produces fluent output. The model 'works'. Then a careful evaluator notices that instruction-following is weaker than expected, system prompts are not respected as tightly as they should be, and behavior is harder to reproduce than the developer expected. The investigation eventually lands on the chat-format handling, and the team learns that the bytes the model sees are not the bytes it was trained on.

The two errors are subtle. Both live entirely in the tokenization layer. Both are invisible without decoding the actual ids and diffing against what apply_chat_template would have produced. Both compound to degrade quality without triggering any alert.

The rest of this explanation walks each error in detail, names the diagnostics that surface them, explains why hand-rolling chat formatting is structurally wrong (not just inconvenient), and closes with the production discipline of routing all formatting through apply_chat_template per model.

Error 1: bare role labels do not produce special role tokens

Modern instruction-tuned models have specific chat formats they were trained on, using special tokens that mark conversation structure. The exact format varies per model family but the principle is universal: role boundaries are marked by tokens the tokenizer knows are special, not by ordinary text strings.

Llama 3 uses <|begin_of_text|> (BOS) plus per-turn <|start_header_id|>{role}<|end_header_id|>\n\n{content}<|eot_id|> plus <|eot_id|> to mark end of turn. ChatML-style models (OpenAI, Mistral instruct, Qwen instruct) use <|im_start|>{role}\n{content}<|im_end|>\n. Both formats use special tokens registered in the tokenizer with single ids that the model learned to recognize as boundaries.

When the developer writes system: and tokenizes it, the tokenizer produces ordinary text tokens: 'system' plus ':' (or some BPE variant of these). The model sees these as content tokens, not as boundary markers. The instruction-tuning behavior that depends on recognizing role transitions degrades because the recognition signal is absent.

The degradation is silent. The model still produces text. It still answers questions. It just answers them with less awareness of where the user's question ended and the system instructions began. System-prompt adherence weakens; multi-turn coherence weakens; instruction-following degrades by 5 to 20 percent on instruction-following benchmarks for a hand-rolled template versus the trained template on the same model.

Error 2: leading-space at the generation boundary

Why both errors are silent and how to detect them

Why hand-rolling is a category error and what apply_chat_template provides

Why option B (the trailing-assistant boundary issue) is the second bug specifically

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

# Wrong: hand-rolled bare-label template
def build_prompt(sys, msg):
    return f'system: {sys}\nuser: {msg}\nassistant:'
prompt = build_prompt(system_text, user_msg)
ids = tokenizer.encode(prompt, add_special_tokens=False)
# Bug 1: 'system:' etc are ordinary text, not special role tokens
# Bug 2: trailing 'assistant:' shifts first-token id distribution

# Right: apply_chat_template
messages = [
    {'role': 'system', 'content': system_text},
    {'role': 'user',   'content': user_msg},
]
ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors='pt',
)
# Diagnostic: decode and compare
print(tokenizer.decode(ids[0]))
# Shows the actual special tokens the model expects

Real products, models, and research that use this idea.

HuggingFace transformers ships apply_chat_template that loads the Jinja template from tokenizer_config.json for Llama 3+, Mistral, Qwen 3.5, and other instruction-tuned models.
Llama 3's documented chat format uses <|begin_of_text|> + <|start_header_id|>{role}<|end_header_id|> + content + <|eot_id|>, none of which appear in a hand-rolled 'system:/user:/assistant:' template.

What an interviewer would ask next. Try answering before peeking at the approach.

QIf you add the right special tokens manually instead of using apply_chat_template, what could still go wrong?

Three things. (1) The model's Jinja template has model-specific spacing and newline rules that hand-coded special-token insertion often misses. (2) The template handles edge cases like multiple system messages, tool messages, and assistant continuations in a specific way that varies per model. (3) Across model upgrades (Llama 3 -> 3.1 -> 3.2), the template can change; loading it dynamically via apply_chat_template adapts, while hand-coded constants do not.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating hand-rolled chat formatting as equivalent to the model's trained chat format, missing both the special role-token mismatch and the leading-space boundary issue.

60 second bullets to scan on the way to the call.

Why bare 'system:'/'user:'/'assistant:' are not the model's special role tokens.
How leading-space changes the first-token id in byte-level BPE.

Primary sources. Browse if you want the original framing.