Which loss function does supervised fine-tuning actually minimize?

**SFT** trains a language model on demonstrations: a prompt and an ideal response. The loss is the same one used during pretraining, applied to the response. At each position in the response, the model produces a probability distribution over its vocabulary, often 50,000 to 200,000 tokens, and predicts the next token. The loss for that position is the negative log probability the model assigned to the true next token. Big number means the model was very surprised; small number means it was confident in the right answer. The total loss for one example is that quantity summed over every response position, then averaged across the batch. Prompt and system tokens are masked out, so they do not contribute to the loss. The model still sees them as context but is not graded on its predictions there. This is **token level cross-entropy**, the standard categorical loss for predicting from a discrete vocabulary. It is the right loss because the target is one specific token at each position, not a continuous value, so a categorical loss makes sense and a regression loss like MSE does not.

**Supervised fine-tuning** minimizes token level cross-entropy on the response tokens. Nothing more exotic. For a single example with response tokens `y_1, ..., y_T` and prefix `x`, the loss is: $$\mathcal{L}_{\text{SFT}} = -\sum_{t=1}^{T} m_t \cdot \log p_\theta\bigl(y_t \mid x, y_{<t}\bigr)$$ The mask `m_t` is 1 for response positions and 0 for prompt and system positions, so only the response contributes. Teacher forcing means at training time the model sees the ground truth prefix when predicting each token, not its own previous predictions. This is exactly the pretraining loss, applied to a smaller curated dataset. That alignment is intentional. Fine-tuning has to match the pretraining objective or the embedding space and the loss disagree, and the model unlearns more than it learns. The other options each describe a different training phase. MSE is for regression, not categorical prediction. The triplet contrastive loss is DPO, which needs paired chosen and rejected responses. The KL divergence to a frozen reference is the PPO regularizer in RLHF, also pair or reward based. SFT is the simpler step that often comes before either of those.