Define the role of a chat template when fine-tuning on (system, user, assistant) data

Flashcard·Easy·4.0 · 0·~30s·Asked atCanva Harvey Notion·Relevant atCoreweave Databricks Lambda Labs Runway

Attempt it

Your answer

A **chat template** is the small piece of code that turns a list of conversation turns into a single string the model can tokenize. A conversation in code looks like a list of dictionaries, one per turn, with a role and a content field. But the model itself never sees that structure. It only sees a sequence of token IDs. So somewhere between the dictionary list and the token sequence, you need to flatten the conversation into text and add markers that tell the model where one turn ends and the next begins. Those markers are special tokens. ChatML wraps each turn with `<|im_start|>` and `<|im_end|>`. Llama-3 uses `<|start_header_id|>`, `<|end_header_id|>`, and `<|eot_id|>`. Mistral uses `[INST]` and `[/INST]`. Each family has its own convention, baked in during pretraining or instruction-tuning. The chat template is the function that applies these markers correctly for a given model. The Hugging Face tokenizer ships a built-in `apply_chat_template` method that does this for you, using a Jinja template stored alongside the tokenizer. The critical rule is: use the template the model was actually trained on. Mix templates and the model gets confused about where turns end.

**A chat template** is the model-specific formatter that turns a structured conversation, typically a list of role and content dictionaries, into the exact token sequence the model expects to see. Its job is to add special role-marker tokens around each turn so the model can tell where one speaker stops and the next begins. Different model families use different conventions. ChatML, the OpenAI-derived convention used by Qwen and others, wraps each turn with `<|im_start|>role` and `<|im_end|>`. Llama-3 uses a header pair `<|start_header_id|>role<|end_header_id|>` plus `<|eot_id|>` at turn end. Mistral and the original Llama-2 use `[INST]` and `[/INST]`. The tokens themselves are added to the vocabulary during pretraining or instruction-tuning, so they tokenize as single IDs rather than character sequences. During fine-tuning you must apply the same template the base model was trained on. If you train on a chat template that the base model never saw, the model has to relearn the role markers from scratch, which wastes capacity and usually fails. Worse, if the template at training time strips or relocates the EOS-like turn-end token, the model never learns to stop generation at the right place. The first symptom in production is a model that rambles past the assistant turn into a hallucinated next user message.

TL;DR

A chat template is the model-specific formatter that wraps each turn in role-marker tokens so the model sees the same turn boundaries at training and inference.

Memory aid

Easy to grasp

Imagine you write a script for a play, but the actor cannot tell who is supposed to speak each line because the script just has names jammed against the dialogue. So you stamp special brackets around every speaker tag and every line ending. Now the actor reads cleanly, knows when one character stops and another starts, and stops at the right moment. The chat template is the rubber stamp that adds those brackets. Different theatre companies use different stamps, and an actor trained with one set of brackets gets confused if you suddenly hand them a script stamped with another set. So whichever model you are training, you have to use the same stamp pattern the model was originally taught to read.

Key concepts

Chat template Special tokens Tokenizer

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

A chat template is the most boring-sounding component in a fine-tuning pipeline and the one most likely to silently ruin a multi-day training run. It is the function that converts a structured conversation, the kind of object your application code naturally produces, into the exact string of characters and special tokens the model was trained to see. Everything downstream, tokenization, loss computation, generation termination, depends on the template being correct for the specific base model you are fine-tuning.

The reason this matters more than it looks is that modern instruction-tuned LLMs were each pretrained or fine-tuned with their own specific template, and the role-marker tokens that template introduces are full first-class members of the vocabulary. The model has gradient-aligned those tokens during training to mean specific things: this is the start of a user turn, this is the end of an assistant response, generation should stop here. If your fine-tuning template uses different markers, you are effectively asking the model to learn new role semantics from scratch on top of the task it is also trying to learn.

This deep dive walks through what a template actually contains, the three major conventions in use today across the 2026 model landscape, the loss-masking interaction that most fine-tuners get wrong, and the silent failure mode where the Hugging Face tokenizer's default template fails to match the base model you loaded.

Anatomy of a chat template

A chat template is functionally a small string-rendering function. It takes a list of message dictionaries, each typically shaped like {'role': 'user', 'content': '...'}, and returns a single string suitable for tokenization.

In the Hugging Face ecosystem the template is stored on the tokenizer as a Jinja2 template and applied via tokenizer.apply_chat_template(messages). The Jinja template iterates over the messages, emits the appropriate role-marker special tokens around each turn's content, optionally adds a system prompt at the head, and may append a generation prompt at the tail (the start of an empty assistant turn that the model will complete).

The special tokens themselves are vocabulary entries added to the tokenizer at pretraining or instruction-tuning time. They tokenize as single token IDs, not as character sequences, which is why you cannot replicate them by just typing the markup string into your data. If you write <|im_start|> as raw text into a tokenizer that has not registered it as a special token, BPE will fragment it into several pieces and the model will not recognise it as a role marker.

A correctly applied template produces output where the role-boundary tokens land at single positions in the token sequence, the content sits between them, and the very last token of any assistant turn is the model's end of turn marker. That last property is the one that determines whether your fine-tuned model will know when to stop talking.

Three template families that cover the 2026 landscape

Loss masking: only the assistant turn should produce gradient

The silent failure mode: tokenizer default template drift

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Hugging Face TRL's SFTTrainer calls tokenizer.apply_chat_template on every example to render the conversation into the model's exact template format.
Llama 4 Maverick fine-tunes use the Llama-3 style header tokens `<|start_header_id|>` and `<|eot_id|>` inherited from the Llama-3 family lineage.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy are role-marker tokens implemented as single special-token IDs rather than plain text strings?

Think about gradient signal and tokenization cost. A single ID gives the model one position to learn role semantics; a multi-character string fragments across BPE pieces and dilutes both signal and attention.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Using the tokenizer's default chat template without checking whether it actually matches the base model you are fine-tuning, then wondering why generations never stop cleanly.

60 second bullets to scan on the way to the call.

Definition of a chat template and the dictionaries to string transformation it performs
Three concrete template formats and which model families use them

Primary sources. Browse if you want the original framing.