Which loss function does supervised fine-tuning actually minimize?
Supervised fine-tuning minimizes token-level cross-entropy on the response tokens, the same objective the base model was pretrained with, just on a smaller curated set.
Imagine teaching someone to copy a famous handwriting sample one letter at a time. At each letter you cover the answer, ask them to guess the next stroke, then reveal the correct one. The lesson scores how surprised they were by the actual letter: very surprised is a big penalty, perfectly expected is a small one. They keep practicing until their guesses match the handwriting closely. SFT works the same way with words. The model reads a prompt, then one word at a time tries to predict the next word in the ideal response. The loss is the surprise score, summed over every word in the response. The system and user prompts are not scored, only the answer is, because the answer is what you are teaching the model to write.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Supervised fine-tuning is so foundational that it is easy to take its loss for granted. The objective is a single line of math, and yet a surprising number of bugs and confused mental models stem from getting that line slightly wrong or confusing it with a different alignment phase.
This question lays out four candidates and asks which one SFT actually minimises. Three are losses from different training phases: a regression loss that does not even fit categorical targets, the DPO preference loss that needs paired data, and the PPO KL regularizer that needs a reference policy. The fourth is plain token level cross-entropy with teacher forcing and loss masking. That fourth answer is correct because SFT inherits its objective from pretraining; the data is different, the loss is not.
The deep dive walks through the mathematics, the masking convention that picks out response tokens, the production gotchas that misalign the mask, and a side by side comparison with the alignment losses that the wrong answers describe. By the end the question should look almost too easy, and the rest of the fine-tuning pipeline should feel like incremental additions to this foundation rather than separate machinery.
The objective and the formula
Take one training example: a prompt x and an ideal response of tokens y_1, ..., y_T. SFT trains the model to assign high probability to each y_t given its prefix.
The per-example loss is the negative log likelihood summed over response positions, with a mask that zeros out non-response tokens:
The mask m_t is 1 when y_t is a response token, 0 otherwise. The probability p_theta is the model's output, computed by passing the prefix through the network and applying a softmax over the vocabulary to the final logits.
Why cross-entropy and not MSE
The target at each position is a discrete token id, effectively a one-hot vector over a vocabulary of 50,000 to 200,000 entries. Cross-entropy is the standard categorical loss for that setting; it has a clean probabilistic interpretation as maximum likelihood. MSE on logits would treat the logit vector as a continuous target, which the target is not. The gradient would be wrong, the loss would be unbounded, and the gradient would push logits toward arbitrary continuous values rather than concentrate probability on the correct token.
Teacher forcing
At training time, the model sees the true prefix when predicting each token, not its own previous prediction. This is teacher forcing, and it is what allows the loss to be computed in parallel across all positions of a sequence in one forward pass. At inference the model generates autoregressively, conditioning on its own outputs, but the loss during training does not.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Hugging Face TRL SFTTrainer wraps PyTorch CrossEntropyLoss with an ignore-index of -100 for prompt positions, the standard recipe used to fine-tune Llama 4 and Mistral on instruction data.
- Axolotl exposes a train_on_inputs flag that toggles whether prompt tokens contribute to the loss, defaulting to false so only response cross-entropy is minimized.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is it important that the SFT loss exactly matches the pretraining loss objective?
Discuss embedding space and loss surface alignment, and explain that a mismatched objective makes the model partially unlearn its pretrained capabilities while trying to satisfy a different target.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Conflating the SFT loss with the DPO pairwise loss or the PPO KL term. SFT is plain next-token cross-entropy on a single ideal response, with no pair, no reference, and no reward model.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.