Spot the bug in this sequence-packing setup
Click any words you think contain an error. Click again to unmark.
Packing without a block-diagonal mask lets examples attend across boundaries, and unmasked loss wastes gradient on prompt and padding. Both bugs stay hidden in the loss curve.
Imagine grading several students' essays stapled into one long packet to save paper. The trick only works if each student can read only their own pages. If you forget the dividers, student three starts quoting student two's essay as if it were their own, and your grading rewards nonsense. The packing setup here forgot the dividers, the default attention mask lets later examples peek at earlier ones. It also grades every word, including the question prompts and blank filler, instead of only the answers you actually want the model to learn. The packet still looks neat and the throughput gauge still climbs, so nobody notices the grades are quietly contaminated.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5 min: packing motivation + default mask contamination + block-diagonal fix + completion-only loss masking + why both bugs are silent + verification on a synthetic pack.
| Concern | Buggy setup | Correct setup |
|---|---|---|
| Attention mask | Default causal over whole pack | Block-diagonal, causal within each example |
| Cross-example attention | Allowed, contaminates context | Blocked at every boundary |
| Loss target | All tokens, prompts and padding included | Response tokens only, prompts and padding masked |
| Symptom | Loss falls, throughput triples, eval degrades | Loss reflects true response learning |
| Implementation | Dense default mask | Variable length kernel with cumulative offsets |
Real products, models, and research that use this idea.
- Hugging Face TRL's SFTTrainer offers packing with example boundary tracking so the attention mask stays block-diagonal rather than naively causal.
- FlashAttention's varlen interface uses cu_seqlens cumulative offsets to pack variable-length examples without letting attention cross segment boundaries.
- Meta's Llama 4 and Mistral fine-tuning recipes pack short instruction examples and rely on segment-aware masks to avoid cross-contamination.
- Axolotl exposes a sample_packing flag that wires up both the block-diagonal mask and completion-only loss masking for SFT runs.
- DeepSpeed and Megatron training stacks pass per-example boundary metadata so packed pretraining and SFT batches keep attention segmented.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does FlashAttention implement block-diagonal masking without materializing a dense mask?
QHow would you detect cross-example contamination after a run has already finished?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Packing examples to save padding but reusing the default causal mask, so tokens attend across example boundaries and the loss covers prompt and padding instead of response tokens only.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.