Spot the bug in this sequence-packing setup

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Quick reference

TL;DR

Packing without a block-diagonal mask lets examples attend across boundaries, and unmasked loss wastes gradient on prompt and padding. Both bugs stay hidden in the loss curve.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine grading several students' essays stapled into one long packet to save paper. The trick only works if each student can read only their own pages. If you forget the dividers, student three starts quoting student two's essay as if it were their own, and your grading rewards nonsense. The packing setup here forgot the dividers, the default attention mask lets later examples peek at earlier ones. It also grades every word, including the question prompts and blank filler, instead of only the answers you actually want the model to learn. The packet still looks neat and the throughput gauge still climbs, so nobody notices the grades are quietly contaminated.

Key concepts

Sequence packing Block-diagonal attention mask Completion-only loss masking

Quick scan · 2–4 min

Everything important, quickly.

5 min: packing motivation + default mask contamination + block-diagonal fix + completion-only loss masking + why both bugs are silent + verification on a synthetic pack.

Side-by-side

Concern	Buggy setup	Correct setup
Attention mask	Default causal over whole pack	Block-diagonal, causal within each example
Cross-example attention	Allowed, contaminates context	Blocked at every boundary
Loss target	All tokens, prompts and padding included	Response tokens only, prompts and padding masked
Symptom	Loss falls, throughput triples, eval degrades	Loss reflects true response learning
Implementation	Dense default mask	Variable length kernel with cumulative offsets

Real-world examples

Real products, models, and research that use this idea.

Hugging Face TRL's SFTTrainer offers packing with example boundary tracking so the attention mask stays block-diagonal rather than naively causal.
FlashAttention's varlen interface uses cu_seqlens cumulative offsets to pack variable-length examples without letting attention cross segment boundaries.
Meta's Llama 4 and Mistral fine-tuning recipes pack short instruction examples and rely on segment-aware masks to avoid cross-contamination.
Axolotl exposes a sample_packing flag that wires up both the block-diagonal mask and completion-only loss masking for SFT runs.
DeepSpeed and Megatron training stacks pass per-example boundary metadata so packed pretraining and SFT batches keep attention segmented.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does FlashAttention implement block-diagonal masking without materializing a dense mask?

Sign in or go Pro to see the approach for this follow-up.

QHow would you detect cross-example contamination after a run has already finished?

Sign in or go Pro to see the approach for this follow-up.

QWhat changes about loss masking when prompts vary wildly in length across the pack?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.