Qwen 3.5 SFT data: fill the special tokens that wrap an assistant turn.
The two special tokens are <|im_start|> and <|im_end|>. Qwen uses the ChatML grammar: open with <|im_start|>assistant, content, close with <|im_end|>.
Picture a stage play script. Every spoken line has a clear opening cue, the actor's name with a colon, and a clear ending cue, a blank line before the next speaker. Without those cues the actors would never know when to stop talking. A chat-tuned model has the same problem. Each assistant turn starts with a small marker that tells the model a reply is beginning, and ends with another marker that tells it to stop. Forget the ending marker in the training data and the model never learns where its turn ends, so at run time it just keeps going. The two markers Qwen uses are short tags that the tokenizer treats as single special words.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Chat templates look like cosmetic string formatting and are actually load-bearing for fine-tuning quality. Qwen 3.5 inherits the ChatML grammar, which wraps every turn between two special tokens: <|im_start|> opens a turn, <|im_end|> closes it. Both are single registered vocabulary ids in the tokenizer.
The full pattern for an assistant turn is <|im_start|>assistant\n{content}<|im_end|>\n. The role name and newline immediately follow the opening token; the closing token plus another newline immediately follow the content. A multi-turn conversation is just this pattern repeated, with system, user, and assistant turns alternating as the conversation requires.
The reason this matters beyond aesthetics. The model learns to stop on the closing <|im_end|> token because that is what appears at the end of every assistant turn in its training data. If your SFT pipeline drops that token (through the wrong template flag, through a careless labels slice, or through hand-assembled strings that bypass apply_chat_template), the model never learns the stop signal. At inference it runs past the expected end of turn and produces rambling output.
This deep dive walks through the ChatML grammar, why the tokens are single registered ids rather than multi-character strings, how to render training data correctly with apply_chat_template, the two common bugs that break stop-token learning, and how to verify a recipe is correct before launching a run.
The ChatML grammar in Qwen 3.5
ChatML is the grammar OpenAI introduced for structured chat conversations and that Qwen adopted starting in the Qwen 1.5 generation. The grammar is small: every turn is wrapped between <|im_start|> and <|im_end|>, with a role name and a newline immediately after the opener.
A full conversation looks like:
<|im_start|>system\n{system_prompt}<|im_end|>\n<|im_start|>user\n{user_message}<|im_end|>\n<|im_start|>assistant\n{assistant_response}<|im_end|>\n
The roles are typically system, user, and assistant. The newline after the role name separates the header from the content, and the newline after <|im_end|> separates one turn from the next.
Both special tokens are registered single-id vocabulary entries in the Qwen tokenizer. The tokenizer treats them atomically: encoding the string <|im_start|> yields a single token id (not a sequence of pipe and angle-bracket character tokens), and decoding that id yields the original string. This is the property that makes them reliable as stop signals: serving stacks configure stops by the integer id, which is robust to whitespace, encoding, and tokenizer version drift.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Qwen 3.5 and the wider Qwen family use ChatML out of the box; the model card on Hugging Face documents the <|im_start|> and <|im_end|> markers as the canonical chat template.
- Hugging Face TRL SFTTrainer routes training data through tokenizer.apply_chat_template by default for Qwen models, which renders the special tokens as single ids correctly.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat changes if you set add_generation_prompt=True versus False, and which is right for training versus inference?
True renders the conversation up to and including the assistant role header (<|im_start|>assistant\n) without closing the turn, leaving the model ready to generate. False renders a complete turn including <|im_end|>. Inference time wants True; training time wants False so the labels contain the stop signal.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating <|im_start|> or <|im_end|> as plain ASCII strings the tokenizer would split. They are single registered vocabulary entries; getting them wrong breaks stop behavior at inference.
60 second bullets to scan on the way to the call.
The two special tokens that wrap a Qwen assistant turn
The full ChatML grammar including the role name and newlines
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.