DeepSeek trains models to predict K tokens per step. Pick how that stays causal.

Attempt it

Multi-token prediction (MTP), used by DeepSeek-V3 and similar designs, trains a model to predict the next K tokens at every position, not just the immediate next one. The key trick is that future tokens are TARGETS, never INPUTS, so causality is preserved. The trunk of the model (the stack of transformer blocks with self-attention) is unchanged from a standard causal decoder. Every position still attends only to itself and earlier positions, the causal mask is exactly the same. On top of the trunk, the model adds K lightweight prediction heads. Head 1 predicts the token at position t+1 (the standard next-token target). Head 2 predicts the token at position t+2. Head 3 predicts t+3. And so on through head K. During training, the loss is the sum of K cross-entropy losses, one per head. The gradient signal is K times denser per forward pass, which is one motivation for MTP: better sample efficiency. At inference, the K heads can drive speculative-style fast decoding from a single model. Head 1 predicts the next token; heads 2 through K propose candidates for positions t+2 through t+K; if those candidates are confirmed by subsequent forward passes, the model effectively decoded K tokens for the cost of one forward. Option A is wrong: relaxing causality would leak the future. Option C is wrong: the architecture is not 'bidirectional within K'. Option D is wrong: DeepSeek-V3 and others ship MTP in production.

Causality is preserved by keeping the trunk's self-attention strictly causal-masked and routing the multi-token prediction signal through K lightweight target heads, never through new input streams. The design has three components: 1. A standard causal transformer trunk. Every token attends only to itself and earlier positions. The mask is identical to a vanilla decoder. 2. K prediction heads on top of the trunk. Each head h takes the trunk hidden state at position t and produces logits for the token at position t + h. Head 1 is the standard next-token head. Heads 2 through K predict progressively further into the future. 3. A summed training loss across all K heads. The training signal is K times denser per forward pass, improving sample efficiency. The critical observation is that future tokens enter the loss as TARGETS, not as INPUTS. The trunk never sees positions t+1 through t+K as input embeddings; those embeddings are still produced one at a time as generation proceeds. The K heads are essentially K supervised classifiers asking 'given what you have read so far, what comes h positions ahead?'. At inference, the K trained heads enable speculative-style fast decoding without a separate draft model. The main head emits the next token; the auxiliary heads emit K-1 speculative candidates that subsequent forward passes verify or reject. DeepSeek-V3 reports 1.8x to 2.6x decoding speedups from this single-model speculative scheme. Option A breaks causality by giving heads future-token attention access. Option C invents a bidirectional within-window design that does not match published architectures. Option D is contradicted by every production MTP deployment in 2026.

Quick reference

TL;DR

MTP preserves causality by keeping the trunk's self-attention causal and adding K lightweight heads that predict future tokens as TARGETS, never as inputs.

Memory aid

Easy to grasp

Imagine a student studying for a quiz where the questions ask 'what is on the next page, and the page after that, and the page after that'. The student is not allowed to peek ahead, they answer from what they have already read. Each answer is graded separately, giving the student more practice per page studied. The student's reading order does not change; they just get more questions per page. That is MTP: the model reads causally as always, but each position is trained to predict several future tokens rather than just the immediate next one.

Key concepts

Causal mask Speculative decoding

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5m: how MTP preserves causality (targets vs inputs), the K-head trunk on top design, the summed training loss, why this enables single-model speculative decoding, and DeepSeek-V3 / V4 production usage.

Real-world examples

Real products, models, and research that use this idea.

DeepSeek-V3 introduced MTP at scale and uses it for both training signal and inference-time speculative decoding.
DeepSeek V4 continues the MTP pattern and reports decoding speedups in the 1.8x to 2.6x range.
Meta has published MTP variants for Llama family research models.
Speculative decoding research more broadly has converged on the 'predict multiple tokens, verify in one forward' pattern.
Production inference stacks like vLLM and TensorRT-LLM have added speculative decoding support that MTP can drive.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the training signal density of MTP help sample efficiency?

QHow does MTP-driven speculative decoding compare to draft-model speculative decoding?

QWhat happens to MTP heads during fine-tuning if you train only on the main head loss?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Confusing 'predicting future tokens' with 'reading future tokens as input'. Targets and inputs are different streams; MTP changes the target stream while keeping the input stream causal.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.