Spot the bug: packed-sequence SFT where the model learns to copy across unrelated examples

Spot the error·Hard·4.0 · 0·~2 min·Relevant atAnthropic Databricks OpenAI

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

The bug is the **attention mask**, not the packing trick itself. Packing several short examples into one long sequence is a good idea for GPU efficiency. The problem is that the standard lower-triangular mask was kept unchanged. A plain causal mask is one big triangle. Every token can attend to every earlier token in the sequence. When the sequence is just one example, that is exactly right. When the sequence contains four or five concatenated examples, every token in example three can now attend to tokens from examples one and two as well as its own. The model is not stupid. During training it notices that those earlier tokens sometimes contain useful patterns, and it learns to incorporate them. At inference it starts copying spans of text from one example into the response of another. That is the cross-example leakage you observed. The fix is a **block-diagonal mask**, also called a segment-causal mask. Within each example the mask is still lower-triangular, but across example boundaries the mask entries are minus infinity so attention is blocked. Modern packing helpers like Unsloth, TRL with packing enabled, and Axolotl with sample_packing=true build this mask automatically. If you assemble packs by hand, the mask must match.

**The mask is the bug.** Packing is fine, the separator token is fine, but the lower-triangular causal mask was inherited from the unpacked recipe and never adjusted. Inside a packed sequence that mask lets every token attend to every earlier position, including positions belonging to unrelated examples sitting upstream in the same pack. The model learns to use that information and at inference it copies spans across example boundaries. The separator token does not save you. It marks the boundary in the input stream, but the attention mechanism has no rule that says 'stop looking before the separator'. Only the mask can enforce that, and a plain causal triangle does not. The correct mask is block-diagonal segment-causal. For each example in the pack, the rows and columns of the mask form a small lower-triangular block; entries outside that block are minus infinity so the softmax assigns them zero weight. Position ids must also reset at each segment boundary so positional encodings stay aligned with the within-example position rather than the global position in the pack. Modern packing helpers handle both pieces. Hugging Face TRL with packing=True, Unsloth's packed dataloaders, and Axolotl with sample_packing=true assemble the block-diagonal mask and reset position ids in one call. FlashAttention 2's varlen interface accepts a cu_seqlens vector that does the equivalent at kernel level without materialising the mask. Hand-rolled packers must replicate both pieces or the cross-example leak resurfaces.

Quick reference

TL;DR

The plain lower-triangular mask lets attention cross example boundaries inside the pack, so the model learns to copy from unrelated earlier examples; the fix is a block-diagonal segment-causal mask.

Memory aid

Easy to grasp

Think of stuffing several short letters into one long envelope to save postage. That part is fine. The problem is the rule you gave the reader: read every previous word in the envelope before answering this question. The reader dutifully scans words from the unrelated letter above, sometimes finds something that sort of fits, and pastes it into the reply. Letters bleed into one another. The fix is not to stop sharing envelopes; it is to change the reading rule. Tell the reader: when you reach a sealed page divider, treat everything above as invisible. Each letter then stays a self-contained unit even though they all share the same envelope. The dividers are real, and the rule needs to respect them.

Key concepts

Sequence packing Block-diagonal mask FlashAttention Cross-example leakage Supervised fine-tuning

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

6 min: identify the plain causal mask as the bug, explain how cross-segment attention leaks during training, prescribe block-diagonal mask plus position-id reset plus varlen kernel use, and verify with a two-example unit test.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face TRL exposes SFTTrainer with packing=True, which builds a block-diagonal mask and resets position ids automatically through FlashAttention 2's varlen path.
Axolotl's sample_packing=true configuration and Unsloth's packed dataloaders both implement segment-causal masking and document the cross-example leak this question asks about.
FlashAttention 2's varlen_func interface takes a cu_seqlens tensor and is the production-grade way to pack without materialising large masks; it powers most modern open-weight LLM training stacks.
Llama 4 and Qwen 3.5 published training recipes both rely on packing with block-diagonal masks at pretraining and fine-tuning scale.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you verify the mask is correct before kicking off a multi-day training run?

QWhat goes wrong if you fix the mask but forget to reset position ids?

QHow does FlashAttention 2's varlen interface avoid materialising the mask at all?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Trusting the separator token to fix the leak. The token marks the boundary in the input but does nothing to block attention across it.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.