DeepSeek trains models to predict K tokens per step. Pick how that stays causal.

MCQ·Medium·4.0 · 0·~1 min·Asked atFractal Analytics Modal Labs Vernacular Ai·Relevant atMeta

Attempt it

Multi-token prediction (MTP), used by DeepSeek-V3 and similar designs, trains a model to predict the next K tokens at every position, not just the immediate next one. The key trick is that future tokens are TARGETS, never INPUTS, so causality is preserved. The trunk of the model (the stack of transformer blocks with self-attention) is unchanged from a standard causal decoder. Every position still attends only to itself and earlier positions, the causal mask is exactly the same. On top of the trunk, the model adds K lightweight prediction heads. Head 1 predicts the token at position t+1 (the standard next-token target). Head 2 predicts the token at position t+2. Head 3 predicts t+3. And so on through head K. During training, the loss is the sum of K cross-entropy losses, one per head. The gradient signal is K times denser per forward pass, which is one motivation for MTP: better sample efficiency. At inference, the K heads can drive speculative-style fast decoding from a single model. Head 1 predicts the next token; heads 2 through K propose candidates for positions t+2 through t+K; if those candidates are confirmed by subsequent forward passes, the model effectively decoded K tokens for the cost of one forward. Option A is wrong: relaxing causality would leak the future. Option C is wrong: the architecture is not 'bidirectional within K'. Option D is wrong: DeepSeek-V3 and others ship MTP in production.

Causality is preserved by keeping the trunk's self-attention strictly causal-masked and routing the multi-token prediction signal through K lightweight target heads, never through new input streams. The design has three components: 1. A standard causal transformer trunk. Every token attends only to itself and earlier positions. The mask is identical to a vanilla decoder. 2. K prediction heads on top of the trunk. Each head h takes the trunk hidden state at position t and produces logits for the token at position t + h. Head 1 is the standard next-token head. Heads 2 through K predict progressively further into the future. 3. A summed training loss across all K heads. The training signal is K times denser per forward pass, improving sample efficiency. The critical observation is that future tokens enter the loss as TARGETS, not as INPUTS. The trunk never sees positions t+1 through t+K as input embeddings; those embeddings are still produced one at a time as generation proceeds. The K heads are essentially K supervised classifiers asking 'given what you have read so far, what comes h positions ahead?'. At inference, the K trained heads enable speculative-style fast decoding without a separate draft model. The main head emits the next token; the auxiliary heads emit K-1 speculative candidates that subsequent forward passes verify or reject. DeepSeek-V3 reports 1.8x to 2.6x decoding speedups from this single-model speculative scheme. Option A breaks causality by giving heads future-token attention access. Option C invents a bidirectional within-window design that does not match published architectures. Option D is contradicted by every production MTP deployment in 2026.

TL;DR

MTP preserves causality by keeping the trunk's self-attention causal and adding K lightweight heads that predict future tokens as TARGETS, never as inputs.

Memory aid

Easy to grasp

Imagine a student studying for a quiz where the questions ask 'what is on the next page, and the page after that, and the page after that'. The student is not allowed to peek ahead, they answer from what they have already read. Each answer is graded separately, giving the student more practice per page studied. The student's reading order does not change; they just get more questions per page. That is MTP: the model reads causally as always, but each position is trained to predict several future tokens rather than just the immediate next one.

Key concepts

Causal mask Speculative decoding

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Multi-token prediction (MTP) sounds like it should violate causality. If a model predicts K tokens at every position, surely it needs to see those K tokens during training? The answer is no, and understanding why is a clean test of how causality is enforced in autoregressive transformers.

Causality lives in the input stream. The model's attention mask determines which positions can read from which others. Targets are unconstrained: the model can be supervised on any function of the inputs, including functions that depend on future positions, as long as the model itself only reads past positions to produce its predictions.

MTP exploits exactly this asymmetry. The trunk's self-attention stays causal: every position attends only to itself and earlier positions. The supervision is multi-target: K separate cross-entropy losses on the K future tokens at each position. The K prediction heads on top of the trunk are essentially K supervised classifiers asking the same internal state different forecasting questions.

This deep dive walks the architecture, the training loss, the inference-time speculative decoding benefit, and the DeepSeek-V3 production deployment that brought MTP into the mainstream.

Targets vs inputs: where causality lives

The single most important conceptual distinction in MTP is between input causality and target structure. They are independent.

Input causality

In an autoregressive transformer, the self-attention mask determines which positions can read from which. The causal mask sets entries above the diagonal to negative infinity before softmax, so position i can only attend to positions j <= i:

M_{ij} = \begin{cases} 0 & i \geq j \\ -\infty & i < j \end{cases}

This is the structural constraint that makes generation possible: the model never reads what it has not yet produced.

Targets

Targets are what the model is supervised against. A standard language model is supervised on the next-token target: at position t, predict the token at t+1. The target is a future token, but the model never SEES that token as input, it predicts it.

The MTP move

MTP keeps input causality exactly as before and multiplies the targets. At position t, the model is supervised on K targets: tokens at positions t+1, t+2, ..., t+K. The trunk reads only past tokens; the K heads predict K future tokens as classification targets.

Causality is about what the model READS, not what it is TRAINED TO PREDICT. MTP changes the second without touching the first.

The architecture in detail

The speculative-decoding bonus

Why option A breaks the design

Where MTP fits in the 2026 landscape

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

DeepSeek-V3 introduced MTP at scale and uses it for both training signal and inference-time speculative decoding.
DeepSeek V4 continues the MTP pattern and reports decoding speedups in the 1.8x to 2.6x range.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the training signal density of MTP help sample efficiency?

Each forward pass produces K supervised targets per position instead of 1, so the model sees more gradient updates per unit of compute. The targets are correlated, so the effective speedup is less than K, but the signal density still improves convergence.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Confusing 'predicting future tokens' with 'reading future tokens as input'. Targets and inputs are different streams; MTP changes the target stream while keeping the input stream causal.

60 second bullets to scan on the way to the call.

How MTP preserves causality: targets vs inputs
The trunk structure: standard causal transformer

Primary sources. Browse if you want the original framing.