Click any words you think contain an error. Click again to unmark.
SFT is next-token cross-entropy over the full sequence. Masking sets prompt-token labels to -100, so only response tokens drive the gradient, not the reverse.
Imagine grading a student who copies the exam question, then writes an answer underneath. You only want to score the answer they wrote, not the question they copied. So you cross out the question with a special mark that says 'ignore this' and grade only the answer. SFT works the same way. The model reads the whole page, prompt plus answer, but you mask the prompt tokens with a -100 label so they earn no penalty. Only the answer tokens shape the model's habits. The buggy paragraph flips this: it claims you mask the answer and grade the question, which would train the model to predict prompts and learn nothing useful about responding.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Spot-the-error questions on SFT loss masking are a favorite senior screen because the mistake is invisible from the outside. A model trained with an inverted mask still produces a falling loss curve, still saves a checkpoint, and still generates fluent text. It is just fluent at the wrong task. So the only defense is a precise mental model of what supervised fine-tuning computes, token by token.
The buggy paragraph hides four errors in three sentences. Two are conceptual mislabels: it calls SFT a contrastive loss, and it claims SFT only sees response tokens. Two are a paired inversion: it masks the response and trains on the prompt, the exact reverse of the standard recipe. None of the four is a typo. Each is a plausible-sounding claim that a candidate who has only skimmed a blog post might repeat.
This deep dive rebuilds the correct picture from the objective outward: what loss SFT actually minimizes, what the model reads versus what it is scored on, how the ignore index implements masking, the off-by-one that lurks in the label shift, and how to articulate the fix crisply under interview pressure.
Error one: SFT is cross-entropy, not a contrastive pair loss
The paragraph opens by calling SFT a contrastive loss between chosen and rejected pairs. That is a category error. Supervised fine-tuning minimizes the same objective as pretraining: next-token cross-entropy against a single target sequence.
The loss for a target of length T is the standard sum over positions:
Notice there is exactly one target sequence per example. No second completion appears anywhere. The model is simply taught to maximize the likelihood of the demonstrated answer, token by token, conditioned on everything to its left.
The moment a method needs a pair of completions, a preferred one and a dispreferred one, you have left SFT and entered preference optimization. DPO, ORPO, and SimPO all consume those pairs and optimize a contrastive or reference-anchored objective rather than raw likelihood. The word 'contrastive' is the giveaway, because cross-entropy never contrasts two candidate outputs against each other. It only pulls probability mass toward the single ground-truth token at each step.
Why does this distinction matter beyond pedantry? The two families sit at different stages of the post-training pipeline. SFT comes first and teaches the model the SHAPE of good answers from demonstrations. Preference methods come after and sharpen the model toward what humans prefer among answers it can already produce. Mislabeling the objective signals that a candidate has not internalized that ordering. So the tell is simple. One target sequence means cross-entropy. A pair of completions means preference learning.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
# labels: prompt tokens masked to -100, response tokens kept
labels = input_ids.clone()
labels[:prompt_len] = -100 # ignore prompt in the loss
# next-token shift handled inside the model: logits[t] predicts labels[t+1]
loss = F.cross_entropy(
logits[..., :-1, :].reshape(-1, vocab),
labels[..., 1:].reshape(-1),
ignore_index=-100,
)Real products, models, and research that use this idea.
- Hugging Face TRL's SFTTrainer sets prompt-token labels to -100 via DataCollatorForCompletionOnlyLM, so only completion tokens contribute to the loss.
- Axolotl and Llama Factory both expose a train_on_inputs flag; leaving it false masks the instruction tokens and trains only on the assistant turn.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is -100 the magic number for the ignore label, and what breaks if you use 0 instead?
Trace it to the framework default ignore_index in cross-entropy. Token id 0 is a real vocabulary entry, often a pad or unknown token, so masking with 0 would silently train on those positions.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing SFT with preference learning, or inverting the mask so prompt tokens drive the gradient. SFT is next-token cross-entropy with prompt labels set to ignore.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.