Define the role of a chat template when fine-tuning on (system, user, assistant) data

Flashcard·Easy·4.0 · 0·~30s

Attempt it

Your answer

A **chat template** is the small piece of code that turns a list of conversation turns into a single string the model can tokenize. A conversation in code looks like a list of dictionaries, one per turn, with a role and a content field. But the model itself never sees that structure. It only sees a sequence of token IDs. So somewhere between the dictionary list and the token sequence, you need to flatten the conversation into text and add markers that tell the model where one turn ends and the next begins. Those markers are special tokens. ChatML wraps each turn with `<|im_start|>` and `<|im_end|>`. Llama-3 uses `<|start_header_id|>`, `<|end_header_id|>`, and `<|eot_id|>`. Mistral uses `[INST]` and `[/INST]`. Each family has its own convention, baked in during pretraining or instruction-tuning. The chat template is the function that applies these markers correctly for a given model. The Hugging Face tokenizer ships a built-in `apply_chat_template` method that does this for you, using a Jinja template stored alongside the tokenizer. The critical rule is: use the template the model was actually trained on. Mix templates and the model gets confused about where turns end.

**A chat template** is the model-specific formatter that turns a structured conversation, typically a list of role and content dictionaries, into the exact token sequence the model expects to see. Its job is to add special role-marker tokens around each turn so the model can tell where one speaker stops and the next begins. Different model families use different conventions. ChatML, the OpenAI-derived convention used by Qwen and others, wraps each turn with `<|im_start|>role` and `<|im_end|>`. Llama-3 uses a header pair `<|start_header_id|>role<|end_header_id|>` plus `<|eot_id|>` at turn end. Mistral and the original Llama-2 use `[INST]` and `[/INST]`. The tokens themselves are added to the vocabulary during pretraining or instruction-tuning, so they tokenize as single IDs rather than character sequences. During fine-tuning you must apply the same template the base model was trained on. If you train on a chat template that the base model never saw, the model has to relearn the role markers from scratch, which wastes capacity and usually fails. Worse, if the template at training time strips or relocates the EOS-like turn-end token, the model never learns to stop generation at the right place. The first symptom in production is a model that rambles past the assistant turn into a hallucinated next user message.

Quick reference

TL;DR

A chat template is the model-specific formatter that wraps each turn in role-marker tokens so the model sees the same turn boundaries at training and inference.

Memory aid

Easy to grasp

Imagine you write a script for a play, but the actor cannot tell who is supposed to speak each line because the script just has names jammed against the dialogue. So you stamp special brackets around every speaker tag and every line ending. Now the actor reads cleanly, knows when one character stops and another starts, and stops at the right moment. The chat template is the rubber stamp that adds those brackets. Different theatre companies use different stamps, and an actor trained with one set of brackets gets confused if you suddenly hand them a script stamped with another set. So whichever model you are training, you have to use the same stamp pattern the model was originally taught to read.

Key concepts

Chat template Special tokens Tokenizer

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: define the dictionaries to string job, walk through three concrete template formats, then cover the EOS and loss-masking traps that catch most teams.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face TRL's SFTTrainer calls tokenizer.apply_chat_template on every example to render the conversation into the model's exact template format.
Llama 4 Maverick fine-tunes use the Llama-3 style header tokens `<|start_header_id|>` and `<|eot_id|>` inherited from the Llama-3 family lineage.
Qwen 3.5 fine-tunes use ChatML markers `<|im_start|>` and `<|im_end|>` because Qwen was instruction-tuned with the ChatML convention.
Mistral and Mixtral-derived fine-tunes still use the older `[INST]` and `[/INST]` markers from the Mistral instruct lineage.
Axolotl and Unsloth both expose chat_template config keys so the user explicitly chooses the template rather than silently inheriting the tokenizer default.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy are role-marker tokens implemented as single special-token IDs rather than plain text strings?

QHow does the chat template interact with loss masking during supervised fine-tuning?

QWhat goes wrong at inference if you fine-tune with the wrong template and serve with the right one?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Using the tokenizer's default chat template without checking whether it actually matches the base model you are fine-tuning, then wondering why generations never stop cleanly.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.