Teacher forcing during SFT: which description nails it?
Teacher forcing means each training position is conditioned on the gold previous tokens, not on the model's own predictions, so the whole sequence is scored in one parallel forward pass.
Imagine teaching a kid to copy a sentence one word at a time. Two ways to do it. The first way, you let the kid write a word, then write the next word based on whatever they wrote, even if their first word was wrong. The second way, you erase any mistake immediately and let them continue from the correct word every time. Teacher forcing is the second way. The kid always sees the right setup, so they learn what comes next from a correct starting point at every step. It also lets you grade every word of the whole sentence at once, instead of waiting for one to be done before moving to the next.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Teacher forcing is the unglamorous mechanism that makes large-scale language model training computationally feasible. The name suggests something subtle, but the idea is mechanical: during training, the previous tokens that condition each prediction are the gold tokens from the training example, not tokens the model produced itself.
The payoff is parallelism. With gold context at every position and the causal attention mask in place, the model can score the entire sequence in one forward pass. Every position simultaneously predicts the next gold token, cross-entropy is summed across positions, and one backward pass updates all the weights. Without this, training would proceed at generation speed, which is roughly two to three orders of magnitude slower per training example.
This deep dive walks through the mechanism, the cooperation with causal attention, the exposure-bias downside that teacher forcing creates, the distinction from related concepts (scheduled sampling, knowledge distillation), and the reason modern alignment recipes accept teacher forcing for SFT while addressing its limits in a separate post-training stage.
The mechanism in concrete terms
Suppose your training example is the assistant response tokens [t1, t2, t3, t4, t5]. With teacher forcing, the model's training inputs at the five positions are [BOS, t1, t2, t3, t4], and the gold targets at the same five positions are [t1, t2, t3, t4, t5]. Every position is being asked to predict the next gold token given the gold prefix.
The transformer processes all five input positions in a single forward pass. The causal attention mask ensures that position i can attend to positions 0 through i but not to positions greater than i. So the prediction at position i depends only on the gold tokens at positions 0 through i-1, never on its own output or on future tokens. The model produces logits at every position in parallel.
The loss is the sum of cross-entropy at each position. PyTorch computes this in a single call by reshaping the logits and labels appropriately. Backprop flows through all positions at once, and the optimiser does one parameter update per batch.
The contrast with the alternative is stark. Without teacher forcing, training would have to proceed step by step: predict t1 from BOS, sample or pick a token, plug it in, predict t2 from BOS and that token, repeat. Each example would take dozens of forward passes instead of one, and the training distribution would be polluted with the model's noisy early outputs.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Every Hugging Face Trainer SFT run on Llama 4 Maverick in 2026 uses teacher forcing by default; the training loop never calls model.generate.
- DeepSpeed and FSDP both rely on teacher-forced parallel forward passes; their throughput numbers assume this mode.
What an interviewer would ask next. Try answering before peeking at the approach.
QWalk through what exposure bias means and how RLHF or DPO compensates for it.
At inference the model conditions on its own outputs, which teacher forcing never simulated. RLHF rolls out actual generations and trains the policy on those rollouts, narrowing the train-inference gap.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing teacher forcing with knowledge distillation. They are orthogonal: teacher forcing is about which previous tokens condition each step, distillation is about what targets the student matches.
60 second bullets to scan on the way to the call.
The definition of teacher forcing in autoregressive sequence training
Why teacher forcing enables parallel per-position scoring
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.