You pad a batch to equal length. How does the model know to ignore the filler tokens?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

A padding token (PAD) is a filler token id appended to short sequences in a batch so every row in the batch reaches the same length, because GPUs operate on rectangular tensors. The attention mask carries a 0 at every pad position and a 1 at every real position; attention multiplies by the mask before the softmax so pad positions get zero weight, and the loss is masked over pad positions during training so they contribute no gradient. Many causal LMs (Llama, GPT family) reuse the EOS id as the pad id; others have a dedicated <pad>. Forgetting to pass the attention mask is the classic bug: the model then attends to the pad tokens and quality drops.

Your answer

A **padding token** (PAD) is a filler token id added to the end (or front) of short sequences in a batch so that every row in the batch has the same length. The reason is mechanical: a GPU runs one matmul per layer on the whole batch, and that matmul needs a rectangular tensor. You cannot send a batch with rows of length 8, 12, and 4 to the model as is; you pad the short ones to 12. The trick that makes pad tokens safe is the **attention mask**. The mask is a binary tensor the same shape as the input ids: 1 at every real token, 0 at every pad. Inside the attention block, pad positions are multiplied by negative infinity before the softmax, which sends their attention weight to zero. The loss is also masked: pad positions are skipped when the cross entropy loss is computed during training, so they produce no gradient. Many causal language models (Llama, the GPT family) do not have a dedicated pad token in their vocabulary and reuse the EOS id for padding. Others ship with a real `<pad>` entry. Either works as long as the attention mask is set correctly. The classic bug is calling the tokenizer with `padding=True` and then forgetting to pass `attention_mask` to the model. The model then attends to the pad ids as if they were real content, and answers degrade in subtle ways.

A **pad token** is a filler token id added to short sequences in a batch so that all rows reach the same length. GPUs run one matmul per layer over the full batch and that matmul requires a rectangular tensor, so unequal sequence lengths have to be padded up to a common length before the batch can be processed. What makes pad tokens safe is not the token id itself; it is the attention mask. The mask is a tensor the same shape as the input ids, with 1 at real positions and 0 at pad positions. Inside the attention block, pad positions are added a large negative number (effectively negative infinity) before the softmax, which drives their post softmax weight to zero. Real tokens never attend to pad. During training, the loss is computed only over real positions, so pad tokens produce no gradient. Both mechanisms together keep the pad ids inert. The specific id used depends on the model. Llama and the GPT family ship without a dedicated pad token and instead reuse the EOS id; the convention is `tokenizer.pad_token = tokenizer.eos_token`. Other tokenizers (BERT, T5, many encoder models) have a real `<pad>` id baked into the vocabulary. The choice does not matter as long as the attention mask is correct. The single most common production bug here is passing padded ids to the model without the attention mask. Many wrappers default `attention_mask` to all ones, which means the model attends to pad as if it were content, and outputs degrade in a way that is hard to debug because the model still produces fluent text.

Quick reference

TL;DR

A pad token is filler added to short rows so the batch is rectangular; the attention mask carries zeros at pad positions so attention assigns them zero weight and the loss ignores them.

Memory aid

Easy to grasp

Imagine a classroom that has to seat everyone in a perfect rectangle of desks. Some rows have eight students, some have three. To make the rectangle work you put cardboard cutouts in the empty desks so the shape is right. The cutouts are pad tokens. Now you give every teacher a seating chart that marks which desks have real students and which have cutouts, and you tell them to only call on real students. That seating chart is the attention mask. If you forget to hand out the chart, the teacher starts asking the cutouts questions, and the lesson goes off the rails.

Key concepts

Padding Attention mask Batching

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: define pad as a batching artifact, explain how the attention mask zeros pad weights inside the softmax, explain loss masking with `-100`, note the EOS reuse convention for causal LMs, and close on the missing attention mask bug.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face `tokenizer(batch, padding=True, return_tensors='pt')` returns both `input_ids` and `attention_mask`; both must flow into the model call.
Llama 3 and Mistral fine tuning recipes set `tokenizer.pad_token = tokenizer.eos_token` and use `padding_side='left'` for generation.
vLLM and SGLang use sequence packing in 2026 production serving so the GPU never spends FLOPs on pad positions across a continuous batch.
Axolotl and Hugging Face TRL training configs default to right padding with the loss masked via `labels` set to `-100` at pad positions.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does a decoder only model require left padding at inference but tolerate right padding during training?

QWhy is reusing the EOS id as the pad id safe for a causal LM?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Tokenizing with `padding=True` and then forgetting to pass `attention_mask` to the model. The model attends to the pad ids as if they were real text and quality drops.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.