Qwen 3.5's chat template wraps each assistant turn as: assistant\n{content}\n. If training data omits the closing token, the model never learns to stop on Qwen's templated stop sequence.
The two tokens are <|im_start|> and <|im_end|>.
Qwen uses a grammar called ChatML for its chat templates. Each turn in a conversation is wrapped between two markers. The opening marker, <|im_start|>, tells the model a new turn is starting. Then comes the role name (system, user, or assistant) and a newline. Then the actual content. Then the closing marker, <|im_end|>, and another newline.
For an assistant turn the full pattern looks like: <|im_start|>assistant\n{content}<|im_end|>\n.
Both markers are single special tokens registered in the tokenizer's vocabulary. They are not regular strings that the tokenizer would break into smaller pieces. If you type them out as plain ASCII without telling the tokenizer they are special, you can end up with a model that sees a string of pipes and angle brackets instead of the actual stop signal.
The reason this matters: if your SFT training data drops the <|im_end|> token from the labels, the model never sees its own stop signal during training. At inference it then refuses to stop where the chat template says it should, and you get rambling output.
**The two tokens are `<|im_start|>` and `<|im_end|>`.**
Qwen 3.5 inherits the ChatML grammar. Every turn in a conversation is wrapped:
```
<|im_start|>{role}\n{content}<|im_end|>\n
```
For an assistant turn specifically the role is `assistant`, so the wrapped string is `<|im_start|>assistant\n{content}<|im_end|>\n`.
Both markers are single registered vocabulary tokens, not multi-character strings the tokenizer would split into pieces. When you call `tokenizer.apply_chat_template(messages, tokenize=True)` against the official Qwen tokenizer, those special tokens render as single ids. If you handcraft training data as raw strings without going through apply_chat_template, you risk the tokenizer treating the literal characters as a sequence of normal tokens, and the model never sees the actual special-token ids.
The practical danger is on the labels side. If you tokenize the prompt correctly with apply_chat_template but then slice the labels in a way that drops the trailing `<|im_end|>` (a common bug is `labels = input_ids[:-1]`), the model never learns to emit the stop token. At inference your stop rule fires on `<|im_end|>`, the model rarely produces one, and generation runs past the expected end of turn. The fix is to keep `<|im_end|>` as a supervised label in the response span and mask only the prompt positions to -100.
Quick reference
TL;DR
The two special tokens are <|im_start|> and <|im_end|>. Qwen uses the ChatML grammar: open with <|im_start|>assistant, content, close with <|im_end|>.
Picture a stage play script. Every spoken line has a clear opening cue, the actor's name with a colon, and a clear ending cue, a blank line before the next speaker. Without those cues the actors would never know when to stop talking. A chat-tuned model has the same problem. Each assistant turn starts with a small marker that tells the model a reply is beginning, and ends with another marker that tells it to stop. Forget the ending marker in the training data and the model never learns where its turn ends, so at run time it just keeps going. The two markers Qwen uses are short tags that the tokenizer treats as single special words.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Training a base model on raw text completion (no chat conversation structure) The chat template is irrelevant when there are no turns; raw next-token prediction does not need <|im_start|> or <|im_end|>.
The Qwen variant you are using has a custom or research chat template Some experimental Qwen releases have used different role names or additional tokens; consult the model card and use apply_chat_template against the exact tokenizer.
Generating data for a different model family (Llama, Mistral, Claude) Each family has its own chat-template grammar; using Qwen's ChatML tokens in a Llama fine-tune produces broken stop behavior.
You are deliberately stripping special tokens for a downstream NLP task that consumes raw text Some pipelines (RAG ingest, search indexing) want the content without the wrapper; strip with skip_special_tokens=True on decode.
Quick scan · 2–4 min
Everything important, quickly.
3 min: name the two tokens, sketch the ChatML grammar, explain the train versus inference flag distinction, then walk through the common labels-slicing bug that breaks stop behavior.
Real-world examples
Real products, models, and research that use this idea.
1Qwen 3.5 and the wider Qwen family use ChatML out of the box; the model card on Hugging Face documents the <|im_start|> and <|im_end|> markers as the canonical chat template.
2Hugging Face TRL SFTTrainer routes training data through tokenizer.apply_chat_template by default for Qwen models, which renders the special tokens as single ids correctly.
3Production serving stacks like vLLM and TGI configure stop tokens for Qwen as the ids of <|im_end|> rather than as string matches, so generation halts deterministically.
4Practitioners building custom multi-turn datasets often round-trip a sample through apply_chat_template plus decode with skip_special_tokens=False to confirm <|im_end|> appears in the right positions.
5Axolotl and Unsloth fine-tuning recipes for Qwen 3.5 expose chat_template='chatml' as a config flag and handle the apply_chat_template call so the user does not have to assemble the string manually.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat changes if you set add_generation_prompt=True versus False, and which is right for training versus inference?
Sign in or go Pro to see the approach for this follow-up.
QHow would you verify your training data has the correct special-token placement before launching a fine-tune?
Sign in or go Pro to see the approach for this follow-up.
QWhy is it dangerous to configure stop tokens by string match rather than by token id at serving time?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating <|im_start|> or <|im_end|> as plain ASCII strings the tokenizer would split. They are single registered vocabulary entries; getting them wrong breaks stop behavior at inference.