DataCollator pads to max_length and sets attention_mask = ones: find what breaks

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Quick reference

TL;DR

Two coupled bugs. The all-ones attention mask makes the model attend to pad tokens; labels equal to input_ids makes cross-entropy grade the model on predicting pad given pad.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a classroom where the teacher hands every student a worksheet with 20 problems, but most students only need to solve 5. The other 15 slots are filled with blank placeholder rows so all worksheets look the same size. Now imagine the teacher grades every row including the placeholders, and treats blank placeholders as if they were real problems the student answered correctly. Two things go wrong. Students who copy a lot of placeholders get suspiciously high scores. And during class discussion, students start studying the placeholders as if they were real content. That is exactly what this collator does: it grades the model on predicting blank padding, and it lets the model spend its attention budget studying those blanks instead of the real question.

Key concepts

Data collator Attention mask Loss masking Padding

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: attention mask role + softmax contamination from all-ones + labels equal to input_ids puts loss on pad + dilution math + correct collator construction + diagnostic signals.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face's DataCollatorForLanguageModeling and the TRL SFTTrainer's default collator both set pad labels to -100 and build the attention mask correctly out of the box.
Axolotl's sample_packing path avoids the pad-heavy regime entirely by concatenating examples to fill max_length, sidestepping both bugs for high-volume SFT runs.
LLaMA-Factory documentation calls out the pad-token loss bug as one of the top three sources of suspiciously low training loss in user-submitted issues.
DeepSeek V4 and Llama 4 fine-tuning notebooks demonstrate completion-only masking as the refinement on top of the basic pad masking, with the prompt span set to -100.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is -100 the conventional ignore_index rather than something like None or 0?

Sign in or go Pro to see the approach for this follow-up.

QHow does this bug manifest differently in a causal versus an encoder model?

Sign in or go Pro to see the approach for this follow-up.

QWhat does the train versus eval loss curve look like when this bug is present?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Fixing only the attention mask and leaving labels equal to input_ids. The loss masking bug is independent and still dominates the gradient signal once the attention bug is gone.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.