DeepSeek trains models to predict K tokens per step. Pick how that stays causal.

Multi-token prediction (MTP), used by DeepSeek-V3 and similar designs, trains a model to predict the next K tokens at every position, not just the immediate next one. The key trick is that future tokens are TARGETS, never INPUTS, so causality is preserved. The trunk of the model (the stack of transformer blocks with self-attention) is unchanged from a standard causal decoder. Every position still attends only to itself and earlier positions, the causal mask is exactly the same. On top of the trunk, the model adds K lightweight prediction heads. Head 1 predicts the token at position t+1 (the standard next-token target). Head 2 predicts the token at position t+2. Head 3 predicts t+3. And so on through head K. During training, the loss is the sum of K cross-entropy losses, one per head. The gradient signal is K times denser per forward pass, which is one motivation for MTP: better sample efficiency. At inference, the K heads can drive speculative-style fast decoding from a single model. Head 1 predicts the next token; heads 2 through K propose candidates for positions t+2 through t+K; if those candidates are confirmed by subsequent forward passes, the model effectively decoded K tokens for the cost of one forward. Option A is wrong: relaxing causality would leak the future. Option C is wrong: the architecture is not 'bidirectional within K'. Option D is wrong: DeepSeek-V3 and others ship MTP in production.

Causality is preserved by keeping the trunk's self-attention strictly causal-masked and routing the multi-token prediction signal through K lightweight target heads, never through new input streams. The design has three components: 1. A standard causal transformer trunk. Every token attends only to itself and earlier positions. The mask is identical to a vanilla decoder. 2. K prediction heads on top of the trunk. Each head h takes the trunk hidden state at position t and produces logits for the token at position t + h. Head 1 is the standard next-token head. Heads 2 through K predict progressively further into the future. 3. A summed training loss across all K heads. The training signal is K times denser per forward pass, improving sample efficiency. The critical observation is that future tokens enter the loss as TARGETS, not as INPUTS. The trunk never sees positions t+1 through t+K as input embeddings; those embeddings are still produced one at a time as generation proceeds. The K heads are essentially K supervised classifiers asking 'given what you have read so far, what comes h positions ahead?'. At inference, the K trained heads enable speculative-style fast decoding without a separate draft model. The main head emits the next token; the auxiliary heads emit K-1 speculative candidates that subsequent forward passes verify or reject. DeepSeek-V3 reports 1.8x to 2.6x decoding speedups from this single-model speculative scheme. Option A breaks causality by giving heads future-token attention access. Option C invents a bidirectional within-window design that does not match published architectures. Option D is contradicted by every production MTP deployment in 2026.