Click any words you think contain an error. Click again to unmark.
Packing without a block-diagonal mask lets examples attend across boundaries, and unmasked loss wastes gradient on prompt and padding. Both bugs stay hidden in the loss curve.
Imagine grading several students' essays stapled into one long packet to save paper. The trick only works if each student can read only their own pages. If you forget the dividers, student three starts quoting student two's essay as if it were their own, and your grading rewards nonsense. The packing setup here forgot the dividers, the default attention mask lets later examples peek at earlier ones. It also grades every word, including the question prompts and blank filler, instead of only the answers you actually want the model to learn. The packet still looks neat and the throughput gauge still climbs, so nobody notices the grades are quietly contaminated.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Sequence packing is a standard throughput optimization for supervised fine-tuning. Short examples waste enormous compute on padding, so you concatenate several of them into a single fixed-length sequence and process the whole thing in one forward pass. The motivation in this setup is correct, and the throughput claim of roughly triple the tokens per step is realistic for a dataset of short instruction examples.
The trouble is that packing changes two invariants that single-example training quietly relied on. With one example per sequence, the default causal mask is exactly right, and computing loss on response tokens is a simple slice. Once you pack, both of those assumptions break, and they break silently. No exception fires, the loss curve still descends, and the throughput gauge confirms the speedup. The corruption only surfaces later, in evaluation.
This spot-the-error has two real bugs and one correct claim. The attention mask is wrong, the loss target is wrong, and the throughput number is right. The danger lives precisely in that asymmetry: the one thing you can see on the dashboard is fine, while the two things you cannot see are broken. This deep dive walks each bug, the fix, and how to verify the setup before committing to an expensive run.
Why the default causal mask contaminates a pack
A causal mask permits each token to attend to itself and every earlier token in the sequence. For a single example this is exactly the autoregressive constraint you want. The model sees only past context when predicting the next token.
The moment you pack multiple examples into one sequence, the word 'earlier' stops respecting example boundaries. A token in the third packed example attends to every token in the first and second examples, because they sit earlier in the same buffer. The model now conditions its prediction on unrelated samples. It learns spurious dependencies that will never exist at inference, where each request arrives as its own sequence.
This is cross-example contamination, and it is invisible in the loss. Attending to extra context often makes next-token prediction slightly easier on the training set, so the training loss can even look better. The damage is a model that has learned to lean on neighbors it will never have at serving time.
The severity scales with how aggressively you pack. With eight examples per sequence, the last example can attend across seven unrelated samples, while the first example is clean. That asymmetry means the contamination is uneven across the batch, which makes it even harder to spot by eyeballing individual losses. Some positions are pristine and others are heavily polluted, and the average smears the two together into a number that looks ordinary.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Concern | Buggy setup | Correct setup |
|---|---|---|
| Attention mask | Default causal over whole pack | Block-diagonal, causal within each example |
| Cross-example attention | Allowed, contaminates context | Blocked at every boundary |
| Loss target | All tokens, prompts and padding included | Response tokens only, prompts and padding masked |
| Symptom | Loss falls, throughput triples, eval degrades | Loss reflects true response learning |
| Implementation | Dense default mask | Variable length kernel with cumulative offsets |
Real products, models, and research that use this idea.
- Hugging Face TRL's SFTTrainer offers packing with example boundary tracking so the attention mask stays block-diagonal rather than naively causal.
- FlashAttention's varlen interface uses cu_seqlens cumulative offsets to pack variable-length examples without letting attention cross segment boundaries.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does FlashAttention implement block-diagonal masking without materializing a dense mask?
Discuss variable length kernels driven by cumulative sequence offsets, where each segment is processed as its own causal block, so memory stays linear in tokens rather than quadratic.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Packing examples to save padding but reusing the default causal mask, so tokens attend across example boundaries and the loss covers prompt and padding instead of response tokens only.
60 second bullets to scan on the way to the call.
Why a default causal mask contaminates packed examples
What a block-diagonal mask blocks and what it preserves
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.