DeepSeek trains models to predict K tokens per step. How does that stay causal?

MCQ·Medium·4.0 · 0·~1 min·Asked atModal Labs Vernacular Ai·Relevant atMeta

Attempt it

Multi-token prediction (MTP), used by DeepSeek-V3 and similar designs, trains a model to predict the next K tokens at every position, not just the immediate next one. Future tokens are TARGETS, never INPUTS, so causality is preserved. The trunk of the model (the stack of transformer blocks with self-attention) is unchanged from a standard causal decoder. Every position still attends only to itself and earlier positions. On top of the trunk, the model adds K lightweight prediction heads. Head 1 predicts the token at position t+1 (the standard next-token target), head 2 predicts t+2, and so on through head K. During training, the loss is the sum of K cross-entropy losses, one per head. The gradient signal is K times denser per forward pass, which improves sample efficiency. At inference, the heads can drive fast speculative decoding from one model. Heads 2 through K propose candidates for positions t+2 through t+K. If later forward passes confirm them, the model decoded K tokens for the cost of one forward. Option A is wrong: relaxing causality would leak the future. Option C is wrong: the architecture is not 'bidirectional within K'. Option D is wrong: DeepSeek-V3 and others ship MTP in production.

Causality is preserved by keeping the trunk's self-attention strictly causal-masked. The multi-token prediction signal goes through K lightweight target heads, never through new input streams. The design has three components: 1. A standard causal transformer trunk. Every token attends only to itself and earlier positions. 2. K prediction heads on top of the trunk. Each head h takes the trunk hidden state at position t and produces logits for the token at position t + h. Head 1 is the standard next-token head. DeepSeek-V3 chains its heads instead: each also reads the token just before its target. 3. A summed training loss across all K heads. The training signal is K times denser per forward pass, improving sample efficiency. Future tokens enter the loss as TARGETS, not as INPUTS. The trunk never sees positions t+1 through t+K as input embeddings; those embeddings are still produced one at a time as generation proceeds. The K heads are K supervised classifiers asking 'given what you have read so far, what comes h positions ahead?'. At inference, the trained heads enable speculative decoding without a separate draft model. The main head emits the next token; the auxiliary heads emit K-1 speculative candidates that later forward passes verify or reject. DeepSeek-V3 reports about 1.8x tokens per second from this single-model speculative scheme. Option A breaks causality by giving heads future-token attention access. Option C's bidirectional within-window design matches no published architecture. Option D is contradicted by production MTP deployments such as DeepSeek-V3.

TL;DR

MTP preserves causality by keeping the trunk's self-attention causal and adding K lightweight heads that predict future tokens as TARGETS, never as inputs.

Memory aid

Easy to grasp

Imagine a student studying for a quiz where the questions ask 'what is on the next page, and the page after that, and the page after that'. The student is not allowed to peek ahead. They answer from what they have already read. Each answer is graded separately, giving the student more practice per page studied. The student's reading order does not change; they just get more questions per page. That is MTP: the model reads causally as always, but each position is trained to predict several future tokens rather than just the immediate next one.

Key concepts

Causal mask Speculative decoding

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Multi-token prediction (MTP) sounds like it should violate causality. If a model predicts K tokens at every position, surely it needs to see those K tokens during training? The answer is no, because of how causality is enforced in autoregressive transformers.

Causality lives in the input stream. The model's attention mask determines which positions can read from which others. Targets are unconstrained: the model can be supervised on any function of the inputs, including functions that depend on future positions. The condition is that the model itself only reads past positions to produce its predictions.

MTP exploits exactly this asymmetry. The trunk's self-attention stays causal: every position attends only to itself and earlier positions. The supervision is multi-target: K separate cross-entropy losses on the K future tokens at each position. The K prediction heads on top of the trunk are K supervised classifiers asking the same internal state different forecasting questions.

Targets vs inputs: where causality lives

The single most important conceptual distinction in MTP is between input causality and target structure. They are independent.

Input causality

In an autoregressive transformer, the self-attention mask determines which positions can read from which. The causal mask sets entries above the diagonal to negative infinity before softmax, so position i can only attend to positions j <= i:

M_{ij} = \begin{cases} 0 & i \geq j \\ -\infty & i < j \end{cases}

This is the structural constraint that makes generation possible: the model never reads what it has not yet produced.

Targets

Targets are what the model is supervised against. A standard language model is supervised on the next-token target: at position t, predict the token at t+1. The target is a future token, but the model never SEES that token as input; it predicts it.

The MTP move

MTP keeps input causality exactly as before and multiplies the targets. At position t, the model is supervised on K targets: tokens at positions t+1, t+2, ..., t+K. The trunk reads only past tokens; the K heads predict K future tokens as classification targets.

Causality is about what the model READS, not what it is TRAINED TO PREDICT. MTP changes the second without touching the first.

The architecture in detail

The speculative decoding bonus

Why option A breaks the design

Where MTP fits in the current landscape

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

DeepSeek-V3 introduced MTP at scale and uses it for both training signal and inference-time speculative decoding.
DeepSeek V4 keeps the MTP setup of DeepSeek-V3, which reports 85-90% acceptance for the second token and about 1.8x tokens per second.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the training signal density of MTP help sample efficiency?

Each forward pass produces K supervised targets per position instead of 1, so the model sees more gradient updates per unit of compute. The targets are correlated, so the effective speedup is less than K, but the signal density still improves convergence.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Confusing 'predicting future tokens' with 'reading future tokens as input'. Targets and inputs are different streams; MTP changes the target stream while keeping the input stream causal.

60 second bullets to scan on the way to the call.

How MTP preserves causality: targets vs inputs
The trunk structure: standard causal transformer

Primary sources. Browse if you want the original framing.