Sequence packing: what it does, and the attention mask gotcha
What is sequence packing in SFT training, why does it improve throughput, and what is the critical attention mask gotcha that distinguishes a correct implementation from a broken one?
Sequence packing concatenates short examples into one max-length sequence to kill padding waste. The gotcha: you need a block-diagonal mask so examples don't attend across boundaries.
Imagine grading short essays, each on its own sheet of paper. Most sheets are tiny but you still feed a full-size sheet through the machine, so you waste paper on blank space. Packing means taping several short essays onto one big sheet so almost no space is wasted. But now a danger appears. If the reader's eyes drift, they might read the end of essay A as if it were the start of essay B and grade them as one muddled piece. So you draw thick black borders between essays. Each essay can only see itself, never its neighbor. In training, that border is a block-diagonal attention mask. Forget it, and the model quietly learns nonsense from glued-together examples while the loss looks perfectly fine.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Sequence packing is one of the highest-leverage throughput tricks in supervised fine-tuning, and also one of the easiest to ship broken. The idea is simple. Most fine-tuning datasets are dominated by short examples, yet training runs with a fixed maximum sequence length. Pad every short example up to that length and you spend most of your GPU budget pushing padding tokens through the forward and backward pass.
The fix is to stop padding each example individually and instead concatenate many short examples into a single full-length sequence. Fewer padding tokens means more real tokens per step, and tokens-per-second is what training cost is actually measured in. On short-example datasets the speedup is routinely 1.5 to 3 times, and on heavily skewed instruction data the gap can be even larger.
The catch is that concatenation changes the meaning of attention. A transformer's default causal mask assumes one continuous document per row. Pack three examples into one row and that assumption breaks: the third example will attend back into the first. The dangerous part is that nothing crashes and the loss curve looks perfect, so the bug ships. That combination of a real speedup and a silent correctness hazard is exactly why the topic separates engineers who have only read about packing from those who have actually shipped it.
This deep dive covers why padding is wasteful, exactly what packing concatenates, the cross-example leakage trap and its block-diagonal fix, the position-id subtlety, how FlashAttention implements packed attention efficiently, and why the whole thing is a favorite senior interview probe.
Why padding wastes so much compute
Training operates on rectangular tensors. Every row in a batch must share the same length, so a batch is padded up to its longest member or to a fixed maximum sequence length. Padding tokens carry no gradient signal, yet they still flow through every matmul in the forward and backward pass, consuming the same FLOPs and memory bandwidth as real tokens.
Consider a dataset where examples are 200, 300, and 800 tokens long and the maximum length is 2048. Pad each example to 2048 and roughly three-quarters of every row is padding. You are paying full price to process tokens that contribute nothing to learning. At scale this is a direct multiplier on your GPU-hour bill, and it is pure waste rather than a tradeoff.
The waste scales with how short and how variable your examples are. Instruction-tuning and chat datasets are exactly this shape, full of short turns, which is why packing is standard practice for fine-tuning rather than a niche optimization. Attention masking on padding suppresses the numerical contribution of those tokens, but it does nothing to recover the wasted compute. The only real cure is to stop emitting padding in the first place, which is precisely what packing does.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | Naive padding | Packing without mask | Packing with block-diagonal mask |
|---|---|---|---|
| Padding waste | High (mostly padding) | Low | Low |
| Throughput | Baseline | High | High |
| Cross-example leakage | None | Yes, silent | None |
| Model correctness | Correct | Broken, looks fine | Correct |
Real products, models, and research that use this idea.
- Hugging Face TRL's SFTTrainer enables packing with a single flag and builds the block-diagonal mask via FlashAttention's cu_seqlens path.
- Axolotl uses sample packing by default for many recipes, materializing segment metadata so attention stays example-local during fine-tuning.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does cross-example leakage stay invisible in the training loss?
Reason about what the loss measures. Next-token prediction can still be plausible when B attends into A, so the loss stays smooth while the learned dependencies are spurious. Only held-out behavioral evals expose it.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Packing examples but reusing the default causal mask, so example B attends into example A. Loss looks normal while the model trains on contaminated cross-example context.
60 second bullets to scan on the way to the call.
Why padding wastes compute on short-example datasets
What packing concatenates and the throughput it buys
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.