Should the system message contribute to the SFT loss?

Attempt it

Quick reference

TL;DR

Mask the system message out of the SFT loss. The model never generates it at serving time, so scoring it just teaches the model to parrot persona templates.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture an acting class where the student has to deliver lines in a play. The director gives the student a character sheet (you are a polite assistant) and a script line from the other actor (please summarise this). The student is graded only on how they say their own line, not on whether they can recite the character sheet or echo the other actor. Grading them on the setup pieces would just teach memorisation of stage directions, which no audience pays to hear. Training is the same. The model gets graded on the assistant turn alone, because that is what gets spoken at serving time. The system and user messages are stage setup, not performance.

Key concepts

Supervised fine-tuning Chat template Loss masking

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: state the answer B, explain why conditioning tokens should not be scored, name the -100 ignore-index mechanism, and end with the per-span loss debugging tip.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face TRL's SFTTrainer with the chat-template formatter masks all non-assistant spans by default in 2026 releases.
Axolotl's train_on_inputs flag toggles this behaviour, and the default false setting is the correct masked behaviour.
Unsloth notebooks on Llama 4 Maverick and Qwen 3.5 fine-tunes use the standard ignore-index convention shipped with PEFT.
DeepSeek V4 fine-tune recipes published in 2026 explicitly recommend response-only scoring for SFT and reserve full-sequence scoring for continued pretraining.
Anthropic's published guidance on building Claude Opus 4.7 character training treats system prompts as conditioning, never as a generation target.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat is the difference between the attention mask and the label mask, and when does each one matter?

Sign in or go Pro to see the approach for this follow-up.

QHow would you debug a fine-tuned chat model that suddenly behaves badly when the system prompt is changed slightly?

Sign in or go Pro to see the approach for this follow-up.

QWhy is response-only scoring the wrong choice for continued pretraining on a domain corpus?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Forgetting to mask the system and user spans, so the model learns to predict the prompt template alongside the answer. The tell is training loss that drops suspiciously fast on shared system text.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.