What is sequence packing in SFT training, why does it improve throughput, and what is the critical attention-mask gotcha that distinguishes a correct implementation from a broken one?
Free · 2 AI evals / day
When you fine-tune on lots of short examples, padding wastes a huge amount of compute. If your longest allowed sequence is 2048 tokens but most examples are 200 to 300 tokens, every batch row is mostly empty padding that still costs forward and backward time.
**Sequence packing** fixes this. It glues several short examples end-to-end into one long sequence that fills the length budget. Far fewer padding tokens means far more real tokens per step, so throughput jumps, often 1.5 to 3 times on short-example data.
There is one big trap. Transformer attention normally lets each token look at every earlier token. Once you concatenate examples A, B, and C, the tokens in B will happily attend back into A, treating A as context for B. That is leakage, and it quietly corrupts what the model learns.
The fix is a block-diagonal attention mask. Each packed example becomes its own island: tokens attend only within their own example, never across the boundary. Libraries like Hugging Face TRL, Axolotl, and Unsloth handle this for you when packing is enabled.
**Sequence packing** concatenates several short training examples into one max-length sequence so you stop paying for padding tokens. With short data padded to a long context window, most of every batch row is padding that still costs forward and backward compute. Packing fills the budget with real tokens, often lifting throughput 1.5 to 3 times.
The critical gotcha is the attention mask. Default causal attention lets every token attend to all prior tokens. Concatenate examples A, B, C naively and B's tokens attend back into A. That is cross-example leakage. The gradient for B is contaminated, and the model learns spurious correlations that have nothing to do with the task.
The fix is a block-diagonal mask. Within each packed example the mask stays causal. Across the boundary it blocks entirely, so each example is its own attention island. Position ids also reset per example.
TRL with packing enabled, Axolotl, and Unsloth all do this correctly. With FlashAttention the mechanism is the cu_seqlens interface, which passes cumulative sequence lengths so packed attention runs efficiently without materializing the mask. Rolling your own packing without the block-diagonal mask is a silent failure: the loss curve looks normal while the model trains on garbage context.
Quick reference
TL;DR
Sequence packing concatenates short examples into one max-length sequence to kill padding waste. The gotcha: you need a block-diagonal mask so examples don't attend across boundaries.
Imagine grading short essays, each on its own sheet of paper. Most sheets are tiny but you still feed a full-size sheet through the machine, so you waste paper on blank space. Packing means taping several short essays onto one big sheet so almost no space is wasted. But now a danger appears. If the reader's eyes drift, they might read the end of essay A as if it were the start of essay B and grade them as one muddled piece. So you draw thick black borders between essays. Each essay can only see itself, never its neighbor. In training, that border is a block-diagonal attention mask. Forget it, and the model quietly learns nonsense from glued-together examples while the loss looks perfectly fine.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Datasets where most examples already fill the context window Padding waste is small, so packing adds complexity for little throughput gain.
A training stack that cannot guarantee a block-diagonal mask Without segment isolation, packing silently corrupts gradients and is worse than plain padding.
Tasks where exact per-example loss accounting is fragile Packing changes per-row token counts, so loss normalization and label masking must be re-verified.
Quick debugging runs where you need simple, inspectable batches Packed batches mix examples per row, making it harder to trace a single sample through the forward pass.
Quick scan · 2–4 min
Everything important, quickly.
4 min: padding waste motivation + what packing concatenates + the cross-example leakage trap + block-diagonal mask plus position-id reset + cu_seqlens implementation + why the failure is silent.
Side-by-side
Aspect
Naive padding
Packing without mask
Packing with block-diagonal mask
Padding waste
High (mostly padding)
Low
Low
Throughput
Baseline
High
High
Cross-example leakage
None
Yes, silent
None
Model correctness
Correct
Broken, looks fine
Correct
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face TRL's SFTTrainer enables packing with a single flag and builds the block-diagonal mask via FlashAttention's cu_seqlens path.
2Axolotl uses sample packing by default for many recipes, materializing segment metadata so attention stays example-local during fine-tuning.
3Unsloth packs sequences for memory-efficient LoRA fine-tunes of Llama 4 and DeepSeek V4 while keeping per-example attention isolation.
4Meta's Llama 4 and DeepSeek V4 pretraining stacks rely on packed sequences with cu_seqlens-style masking to keep GPU utilization high.
5FlashAttention 2 and 3 expose varlen kernels keyed on cumulative sequence lengths, the standard primitive every packing library calls.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does cross-example leakage stay invisible in the training loss?
Sign in or go Pro to see the approach for this follow-up.
QWhy must position ids reset per segment even with a correct block-diagonal mask?
Sign in or go Pro to see the approach for this follow-up.
QHow does FlashAttention's cu_seqlens avoid materializing a dense mask?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Packing examples but reusing the default causal mask, so example B attends into example A. Loss looks normal while the model trains on contaminated cross-example context.