Spot the masking bug in this packed-sequence training setup

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Quick reference

TL;DR

A plain causal mask lets each example attend backward into earlier packed examples, leaking signal. Layer a block-diagonal document mask on top, or use FlashAttention-2's cu_seqlens API.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine printing five different children's stories on the same long roll of paper, end to end, to save paper. You then hand the roll to a student and ask them to predict the next word at any point, only looking to the left of where they are. Without any marks separating the stories, the student happily looks left and gets to read the end of story 4 as 'context' for the beginning of story 5, which is completely the wrong context. The fix is to draw a thick vertical line between every story so the student knows they can only look left as far as the most recent line. Packed-sequence attention works the same way: the lines between examples must be enforced, not just inferred.

Key concepts

Causal mask Document mask (block-diagonal)Packed sequences FlashAttention

Quick scan · 2–4 min

Everything important, quickly.

Explain the causal-only mask's blind spot at example boundaries, motivate the document mask as a block-diagonal overlay, show how the masks compose via elementwise AND, walk through FlashAttention-2's cu_seqlens implementation, and close with the production checklist for packed-sequence training.

Code example

python

import torch
from flash_attn import flash_attn_varlen_func

# Three examples packed into one sequence of length 17 + 25 + 47 = 89.
example_lengths = [17, 25, 47]
cu_seqlens = torch.tensor(
    [0] + list(torch.tensor(example_lengths).cumsum(0).tolist()),
    dtype=torch.int32,
)  # [0, 17, 42, 89]

# Q, K, V are packed: shape (total_tokens, n_heads, d_head)
total_tokens = sum(example_lengths)  # 89
n_heads, d_head = 8, 64
Q = torch.randn(total_tokens, n_heads, d_head, dtype=torch.float16, device='cuda')
K = torch.randn(total_tokens, n_heads, d_head, dtype=torch.float16, device='cuda')
V = torch.randn(total_tokens, n_heads, d_head, dtype=torch.float16, device='cuda')

# FlashAttention-2 applies causal + block-diagonal (per cu_seqlens) masking.
output = flash_attn_varlen_func(
    Q, K, V,
    cu_seqlens_q=cu_seqlens, cu_seqlens_k=cu_seqlens,
    max_seqlen_q=max(example_lengths), max_seqlen_k=max(example_lengths),
    causal=True,
)
# output has shape (total_tokens, n_heads, d_head)
# Each example's tokens attend only to earlier tokens in the same example.

Real-world examples

Real products, models, and research that use this idea.

FlashAttention-2 (Dao 2023) introduced the cu_seqlens API specifically to support efficient packed-sequence training.
Megatron-LM, DeepSpeed, and most production training stacks support packed sequences with document masking via cu_seqlens or equivalent.
Llama 4 Maverick and similar 2026 frontier models train with packed sequences at high density (8x to 16x) using document masking.
Mistral and Qwen 3.5 use packed-sequence training during pretraining and SFT phases.
PyTorch's FlexAttention API (2024) exposes arbitrary mask functions including block-diagonal patterns as first-class inputs.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does FlashAttention-2's cu_seqlens API avoid materializing the (T, T) document mask?

Sign in or go Pro to see the approach for this follow-up.

QWhat goes wrong if you reset positional encoding at example boundaries but forget the document mask?

Sign in or go Pro to see the approach for this follow-up.

QWhy does packed training give such a large throughput win when it requires this extra masking machinery?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.