Teacher forcing in pretraining: what is fed at each step?

Teacher forcing means the model uses the true prior tokens from the training text when predicting token t. So if the sentence is "The cat sat", the model predicts "cat" while seeing "The", then predicts "sat" while seeing "The cat" from the dataset. It does not feed its own sampled prediction back during this training step. This matters because it keeps gradients stable early and enables parallel training with masked attention. The model still learns an autoregressive objective, but with clean context. At inference, the process changes: now the model must condition on its own generated history.

Teacher forcing is the train-time conditioning scheme for autoregressive pretraining: at position t, the model predicts token `x_t` using the ground-truth prefix `x_<t` from data. That gives two practical wins. First, optimization is more stable because context does not drift from early wrong predictions. Second, with causal masking, all positions in a sequence can be computed in parallel in one forward pass. The objective remains next-token cross-entropy: $$\mathcal{L} = -\sum_t \log p(x_t\mid x_{<t})$$ The key boundary is train vs inference. Teacher forcing is train-time only. At inference, predictions become the next-step inputs.