DataCollator pads to max_length and sets attention_mask = ones: find what breaks
Click any words you think contain an error. Click again to unmark.
Two coupled bugs. The all-ones attention mask makes the model attend to pad tokens; labels equal to input_ids makes cross-entropy grade the model on predicting pad given pad.
Picture a classroom where the teacher hands every student a worksheet with 20 problems, but most students only need to solve 5. The other 15 slots are filled with blank placeholder rows so all worksheets look the same size. Now imagine the teacher grades every row including the placeholders, and treats blank placeholders as if they were real problems the student answered correctly. Two things go wrong. Students who copy a lot of placeholders get suspiciously high scores. And during class discussion, students start studying the placeholders as if they were real content. That is exactly what this collator does: it grades the model on predicting blank padding, and it lets the model spend its attention budget studying those blanks instead of the real question.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Pad handling in a data collator is one of those areas where two small omissions interact to produce a run that looks healthy on the surface and is quietly broken underneath. The setup in the question is realistic and dangerous. A custom collator pads every example to a fixed max_length, fills the unused slots with the pad token, sets the attention mask to all ones, and uses labels equal to input_ids. Each individual decision sounds reasonable, but together they tell the model that the pad slots are real content in both the attention pathway and the loss pathway. The result is a run with suspiciously low training loss, mediocre generation quality, and an eval curve that quietly tells you something is wrong if you bother to plot it.
The bug is best understood as two independent failures that share a root cause. The attention mask controls what the softmax pays attention to. The label tensor controls what cross-entropy grades. Pad tokens need to be excluded from both, but they are excluded by different mechanisms. The attention path uses a 0 in the mask to push pad positions to negative infinity inside the softmax. The loss path uses -100 in the labels to tell cross-entropy to skip those positions in the average. The collator in the question does neither.
The practical consequence is that the model is being trained on a corrupted objective. Most of the cross-entropy gradient comes from predicting pad given pad, which is trivially easy and dominates the per-token average. Most of the attention budget at later sequence positions is spent mixing pad embeddings into the hidden state. The model learns to be very good at the pad-prediction task, mediocre at the real generation task, and the headline training loss metric is uninformative. This deep dive walks through each failure mode mechanically, shows how to diagnose them from training curves, and gives the exact collator pattern that fixes both.
The attention mask: why all-ones contaminates every layer
The attention mask is consumed by the attention layer right before the softmax. A standard implementation adds a large negative number, often negative infinity, to the pre-softmax scores at positions where the mask is 0. The exponential of negative infinity is zero, so those positions contribute nothing to the softmax denominator and receive no probability mass.
When the mask is all ones, no positions are pushed to negative infinity. Every key participates in the softmax, including the pad slots. The pad token has an embedding like any other token, which means there is a learnable vector that gets queried during attention. The softmax distributes probability across all keys including pad. The value-side weighted sum then mixes pad value vectors into the per-query output. The residual stream at every layer carries a small but nonzero contribution from pad.
Gradients flow back the same way. The pad embedding receives updates on every backward pass, slowly drifting into whatever direction the loss demands. After a few thousand steps, the pad embedding is a learned junk vector that the model has implicitly trained against. Worse, the pad position influences nearby real-token representations, so the network's understanding of the real content is itself a function of how much pad happened to be in the batch.
A causal mask muddies but does not fix the picture. The lower-triangular structure means a real token at position i never attends to a future pad at position j > i. But a real token at position i does attend to all previous pads if any exist, which is the standard situation for left-padded batches and for any batch where short examples are padded on either side. Right padding plus causal masking limits the damage but does not eliminate it, because the loss computation still requires forward passes that touch pad positions in intermediate layers.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Hugging Face's DataCollatorForLanguageModeling and the TRL SFTTrainer's default collator both set pad labels to -100 and build the attention mask correctly out of the box.
- Axolotl's sample_packing path avoids the pad-heavy regime entirely by concatenating examples to fill max_length, sidestepping both bugs for high-volume SFT runs.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is -100 the conventional ignore_index rather than something like None or 0?
PyTorch's cross_entropy implementation uses -100 as the sentinel for skipped positions because vocabulary indices are non-negative. A negative value is unambiguous and avoids confusing skipped positions with a real token ID of zero, which is often the unknown or pad token in practice.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Fixing only the attention mask and leaving labels equal to input_ids. The loss masking bug is independent and still dominates the gradient signal once the attention bug is gone.
60 second bullets to scan on the way to the call.
Role of the attention mask in the softmax denominator at pad positions
Why an all-ones mask contaminates the residual stream
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.