Why does completion-only loss usually outperform full-sequence loss on the same SFT data?

MCQ·Medium·4.0 · 0·~1 min·Relevant atDatabricks Meta OpenAI

Attempt it

**Completion-only loss** masks out the prompt tokens during loss computation. Only the response tokens contribute to the gradient. Think about a typical instruction example: the user writes 'Explain photosynthesis to a fifth grader' and the response is a paragraph of explanation. The prompt is often 30 to 70 percent of the sequence length. Full-sequence loss trains the model to predict every token, including the prompt tokens. That means a chunk of every gradient step is spent teaching the model to predict 'Explain photosynthesis...' from previous tokens. But the model will never need to generate the prompt at inference. The user types the prompt; the model generates the response. Completion-only loss fixes this. It masks the prompt positions so they contribute zero gradient. Every optimizer step now spends its full budget on the tokens the model actually has to produce. The payoff is faster learning on the target distribution. Same number of steps, same data, but the gradient signal is concentrated rather than diluted. The other options in the question blame mechanisms (learning rate, EOS, FlashAttention) that loss masking does not change. Only the loss aggregation changes.

**Completion-only loss** is loss masking that zeros out the prompt-token positions, so cross-entropy is computed only over the response span. The forward pass is unchanged; the model still attends across the full sequence. Only the gradient aggregation differs. The win comes from gradient budget allocation. In a typical SFT example, the prompt is anywhere from 30 to 70 percent of the sequence. Full-sequence loss spreads the gradient across all tokens equally, so a large fraction of every step is spent teaching the model to predict text the user wrote, text the model will never need to generate at inference. Completion-only re-routes that same budget onto the response tokens, which are exactly what the model has to learn to produce. The other options confuse mechanism with consequence. Loss masking does not change the learning rate (option B). It does not affect EOS handling (option C); EOS is typically inside the response span and gets the same treatment under both schemes. FlashAttention still processes the full sequence on the forward (option D); only the loss is masked, not the attention computation. In practice, completion-only is the default in TRL's `SFTTrainer`, Axolotl, and LLaMA-Factory. Switching from full-sequence to completion-only typically improves downstream evals by 1 to 3 points at the same training budget on instruction data with verbose prompts.

Quick reference

TL;DR

Completion-only loss masks the prompt so gradient flows only into response tokens. Full-sequence loss wastes gradient teaching the model to predict text the user already typed.

Memory aid

Easy to grasp

Imagine teaching a student to write replies to emails. If you grade them on every word, including the original email they received, half their effort goes into copying back the sender's words instead of writing the reply. So you cover the original email with a sticky note and grade only the reply they actually wrote. Same homework, same time, same effort, but now every red pen mark falls on the part that matters. The student gets better at replies twice as fast. Completion-only loss is exactly that sticky note: it hides the prompt from the grading pen so every correction lands on the response.

Key concepts

Supervised fine-tuning Loss masking Cross-entropy loss Instruction tuning

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: loss masking mechanism + identical forward and attention + prompt-fraction sizing + gradient reallocation + distractor refutation + multi-turn handling + when full-sequence still helps.

Real-world examples

Real products, models, and research that use this idea.

TRL's `SFTTrainer` defaults to completion-only loss via `DataCollatorForCompletionOnlyLM`, masking prompt token labels to -100 before cross-entropy.
Axolotl exposes a `train_on_inputs: false` flag (the default) that implements completion-only masking on the chat template's response span.
LLaMA-Factory's SFT recipes apply completion-only by default; switching to full-sequence requires an explicit config override.
OpenAI and Anthropic fine-tuning APIs implicitly use completion-only loss; the documentation specifies that loss is computed on assistant turns only.
Llama 4 and Qwen 3.5 instruction-tuning ablations published in 2026 report 1 to 3 point gains on MT-Bench and IFEval from switching full-sequence to completion-only on the same SFT mix.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does completion-only often help more on long-prompt datasets than on short-prompt datasets?

QWhen might full-sequence loss actually be preferable to completion-only?

QHow does completion-only interact with multi-turn chat templates?

QDoes completion-only loss reduce overfitting on small datasets?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming completion-only changes the forward pass or attention. It does not. The full sequence still runs through the model; only the loss aggregation masks out prompt-token positions before backward.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.