Should the system message contribute to the SFT loss?
Mask the system message out of the SFT loss. The model never generates it at serving time, so scoring it just teaches the model to parrot persona templates.
Picture an acting class where the student has to deliver lines in a play. The director gives the student a character sheet (you are a polite assistant) and a script line from the other actor (please summarise this). The student is graded only on how they say their own line, not on whether they can recite the character sheet or echo the other actor. Grading them on the setup pieces would just teach memorisation of stage directions, which no audience pays to hear. Training is the same. The model gets graded on the assistant turn alone, because that is what gets spoken at serving time. The system and user messages are stage setup, not performance.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Loss masking in supervised fine-tuning is one of those topics that sounds trivial until it bites a production run. The question is simple to state: when an SFT example contains a system message, a user turn, and an assistant turn, which tokens contribute to the training loss? The answer is equally simple, only the assistant turn, but the reasoning behind it is worth unpacking because the wrong choice is a common bug.
The rule is not about the system message specifically, it is about what the model has to generate at serving time. Anything the model conditions on but does not produce should not contribute gradient. The system message falls into that category, as does the user turn, as do retrieved chunks in RAG fine-tunes and tool schemas in agent fine-tunes.
This deep dive walks through the mechanism (the -100 ignore-index convention), why scoring conditioning tokens is harmful (templated overfit), how to debug a suspected mask leak (per-span loss), and the wider generalisation of the rule beyond chat models.
What SFT is actually teaching
Supervised fine-tuning teaches the model a conditional distribution. Given the prior conversation, generate the assistant turn that follows. The training signal is cross-entropy between the model's next-token predictions and the gold assistant tokens. Conceptually it is no different from next-token prediction in pretraining, but the loss is restricted to the assistant span instead of every position.
Why the restriction? Because at serving time, the model never has to generate the system message or the user turn. Those arrive as inputs from the application or the user. The only thing the model produces is the assistant response, so the only thing worth scoring during training is the assistant response.
If you score the conditioning tokens too, the model spends some of its capacity learning to predict the system template and the user phrasing. That capacity is essentially wasted because the model will never need it. Worse, if many training examples share the same system prompt, the model overfits hard to it. Training loss drops fast because the templated tokens become trivial to predict, and you walk away thinking the model trained well when in fact most of the loss reduction came from memorising the wrapper, not from learning the response style.
This is the failure mode option A bakes in. Option C invents a length rule that does not exist; the conditioning versus output distinction is absolute, not length-dependent. Option D conflates two layers that are independent.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Hugging Face TRL's SFTTrainer with the chat-template formatter masks all non-assistant spans by default in 2026 releases.
- Axolotl's train_on_inputs flag toggles this behaviour, and the default false setting is the correct masked behaviour.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat is the difference between the attention mask and the label mask, and when does each one matter?
Attention mask controls which tokens can be looked at during forward; label mask (or ignore index) controls which positions contribute to the loss. The first changes the representation, the second changes the gradient.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forgetting to mask the system and user spans, so the model learns to predict the prompt template alongside the answer. The tell is training loss that drops suspiciously fast on shared system text.
60 second bullets to scan on the way to the call.
Definition of supervised fine-tuning and what it teaches the model to generate
Why the system message is conditioning and not a generation target
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.