Describe what the SFT loss computes for a single (prompt, response) training example. What does 'completion only loss' or 'prompt masking' mean, and what failure mode does it prevent?
SFT is next-token cross-entropy over the prompt-plus-response sequence. Prompt masking sets prompt targets to -100 so only response tokens train the model.
Picture a fill-in-the-blank worksheet where the question is already printed and you must write the answer. SFT grades the model on every blank it could fill. But you do not want credit for re-copying the printed question; you already have it. So a teacher draws a line through the question part and only marks the answer you wrote. Prompt masking is that line: it tells the grader to skip the question tokens and score only the response tokens. The model still reads the full question to understand context. It earns a grade only on the words it was supposed to generate. That way all of the learning effort goes into shaping good answers, not into memorising prompts.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Supervised fine-tuning sounds like it should have an exotic objective. It does not. The loss that shapes an instruction-following model is the same next-token cross-entropy that pretrained it. The key is not the formula but which tokens you let the formula see.
Two things matter. First, the loss itself, including the often-forgotten one-position shift between predictions and targets. Second, prompt masking, the small bookkeeping trick that decides whether your gradient teaches behaviour or wastes itself reconstructing inputs the model is always handed.
The masking is a single boolean flag in most frameworks. Yet getting it wrong, or misunderstanding what it does, produces slow convergence, silent off-by-one bugs, and models that train on the wrong spans.
The loss: next-token cross-entropy
SFT trains on a sequence formed by concatenating a prompt with its target response, tokenised into one flat array. The model is autoregressive, so at each position it produces a probability distribution over the vocabulary for the next token.
Cross-entropy scores each of those distributions against the true next token. For a single position, the penalty is the negative log probability the model assigned to the correct token. Lower probability on the right token means a larger penalty and a stronger gradient pushing that token up.
The standard form, where the target is a one-hot vector over the vocabulary, is:
Summed across every scored position and reduced to a scalar, this is the entire objective. It is identical to the pretraining loss. Nothing about SFT changes the math; the dataset changes, and the set of scored positions changes.
Because the loss is per token and additive, every scored position pulls the weights independently toward its target. There is no holistic 'response quality' term, no notion of the answer being good as a whole. The model learns, token by token, to make the observed continuations more probable in context. Any higher-level behaviour you observe, a consistent tone or a reliable format, emerges from many local next-token corrections accumulating across the dataset. That is why data quality and coverage dominate SFT outcomes far more than any clever loss variant.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import torch.nn.functional as F
# logits: [B, T, V]; labels: [B, T] with -100 on prompt tokens
shift_logits = logits[:, :-1, :].contiguous()
shift_labels = labels[:, 1:].contiguous()
loss = F.cross_entropy(
shift_logits.view(-1, shift_logits.size(-1)),
shift_labels.view(-1),
ignore_index=-100, # prompt tokens skipped
)| Aspect | Full-sequence loss | Completion-only loss |
|---|---|---|
| Tokens scored | Prompt plus response | Response only |
| Prompt label | True next token | -100 (ignore_index) |
| Gradient use | Partly spent on prompts | All on response behaviour |
| Small-dataset effect | Can help with long prompts and short responses | Standard default; gain depends on data |
| Large-dataset gap | Shrinks (response dominates) | Still default, near free |
Real products, models, and research that use this idea.
- Hugging Face TRL's SFTTrainer computes completion-only loss by default for prompt-completion datasets (SFTConfig(completion_only_loss=True)). Setting assistant_only_loss=True masks everything but assistant turns in conversational data.
- Axolotl exposes train_on_inputs: false to mask instruction tokens, a common setting for fine-tuning open chat models such as Llama and Qwen.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the choice between token-mean and sample-mean reduction change what SFT optimises?
Token-mean weights gradient by response length, so verbose examples dominate. Sample-mean normalises per example. Discuss how this interacts with sequence packing and length-imbalanced datasets.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying SFT uses a special loss. It is plain next-token cross-entropy, the same objective as pretraining. The only change is which tokens contribute.
60 second bullets to scan on the way to the call.
SFT loss is next-token cross-entropy, identical to pretraining
The one-position shift between logits and labels
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.