You pad a batch to equal length. How does the model know to ignore the filler tokens?

Flashcard·Easy·4.0 · 0·~30s·Asked atRephrase Ai Synthesia Tencent·Relevant atMeta

Attempt it

A padding token (PAD) is a filler token id appended to short sequences in a batch so every row in the batch reaches the same length, because GPUs operate on rectangular tensors. The attention mask carries a 0 at every pad position and a 1 at every real position; attention multiplies by the mask before the softmax so pad positions get zero weight, and the loss is masked over pad positions during training so they contribute no gradient. Many causal LMs (Llama, GPT family) reuse the EOS id as the pad id; others have a dedicated <pad>. Forgetting to pass the attention mask is the classic bug: the model then attends to the pad tokens and quality drops.

Your answer

A **padding token** (PAD) is a filler token id added to the end (or front) of short sequences in a batch so that every row in the batch has the same length. The reason is mechanical: a GPU runs one matmul per layer on the whole batch, and that matmul needs a rectangular tensor. You cannot send a batch with rows of length 8, 12, and 4 to the model as is; you pad the short ones to 12. The trick that makes pad tokens safe is the **attention mask**. The mask is a binary tensor the same shape as the input ids: 1 at every real token, 0 at every pad. Inside the attention block, pad positions are multiplied by negative infinity before the softmax, which sends their attention weight to zero. The loss is also masked: pad positions are skipped when the cross entropy loss is computed during training, so they produce no gradient. Many causal language models (Llama, the GPT family) do not have a dedicated pad token in their vocabulary and reuse the EOS id for padding. Others ship with a real `<pad>` entry. Either works as long as the attention mask is set correctly. The classic bug is calling the tokenizer with `padding=True` and then forgetting to pass `attention_mask` to the model. The model then attends to the pad ids as if they were real content, and answers degrade in subtle ways.

A **pad token** is a filler token id added to short sequences in a batch so that all rows reach the same length. GPUs run one matmul per layer over the full batch and that matmul requires a rectangular tensor, so unequal sequence lengths have to be padded up to a common length before the batch can be processed. What makes pad tokens safe is not the token id itself; it is the attention mask. The mask is a tensor the same shape as the input ids, with 1 at real positions and 0 at pad positions. Inside the attention block, pad positions are added a large negative number (effectively negative infinity) before the softmax, which drives their post softmax weight to zero. Real tokens never attend to pad. During training, the loss is computed only over real positions, so pad tokens produce no gradient. Both mechanisms together keep the pad ids inert. The specific id used depends on the model. Llama and the GPT family ship without a dedicated pad token and instead reuse the EOS id; the convention is `tokenizer.pad_token = tokenizer.eos_token`. Other tokenizers (BERT, T5, many encoder models) have a real `<pad>` id baked into the vocabulary. The choice does not matter as long as the attention mask is correct. The single most common production bug here is passing padded ids to the model without the attention mask. Many wrappers default `attention_mask` to all ones, which means the model attends to pad as if it were content, and outputs degrade in a way that is hard to debug because the model still produces fluent text.

TL;DR

A pad token is filler added to short rows so the batch is rectangular; the attention mask carries zeros at pad positions so attention assigns them zero weight and the loss ignores them.

Memory aid

Easy to grasp

Imagine a classroom that has to seat everyone in a perfect rectangle of desks. Some rows have eight students, some have three. To make the rectangle work you put cardboard cutouts in the empty desks so the shape is right. The cutouts are pad tokens. Now you give every teacher a seating chart that marks which desks have real students and which have cutouts, and you tell them to only call on real students. That seating chart is the attention mask. If you forget to hand out the chart, the teacher starts asking the cutouts questions, and the lesson goes off the rails.

Key concepts

Padding Attention mask Batching

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Padding looks like a trivial implementation detail until you trace what would happen without it. GPUs cannot process a batch of unequal length sequences directly; the matmul that the attention block runs has to be rectangular. Padding makes the batch rectangular, and the attention mask makes the padding invisible. Together they are the bridge between irregular real text and the regular tensors the hardware needs.

This explanation walks through what a pad token is, what the attention mask does inside the attention block, how the loss is masked during training, the convention of reusing EOS as PAD in causal LMs, and the production bugs that come up when one of these layers is missing or wrong.

Why padding exists at all

Real text comes in unequal lengths. A batch of three sentences might tokenize to lengths 8, 12, and 4. The GPU runs the attention block as one big matrix multiplication over the whole batch, and that matmul needs the input tensor to be rectangular. The shape has to be [batch_size, seq_len, hidden] with a single seq_len for the whole batch.

The fix is to choose a common length (usually the longest in the batch) and fill the shorter rows up to that length with a filler token id. That filler id is the pad token. After padding, all three rows are length 12 and the tensor is rectangular.

The pad id itself is arbitrary in the sense that the model gives it no special semantic meaning. What makes the id behave as filler is everything that happens around it: the attention mask, the loss mask, and the way the runtime feeds both into the model. Without those, the pad id is just another token the model treats as content.

How the attention mask makes pad inert

How the loss mask handles pad during training

Why causal LMs reuse EOS as PAD

Production bugs and the cost of getting padding wrong

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Hugging Face `tokenizer(batch, padding=True, return_tensors='pt')` returns both `input_ids` and `attention_mask`; both must flow into the model call.
Llama 3 and Mistral fine tuning recipes set `tokenizer.pad_token = tokenizer.eos_token` and use `padding_side='left'` for generation.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does a decoder only model require left padding at inference but tolerate right padding during training?

Inference generates from the last position in the sequence, so the last real token must be at the last index; right padding would put pad there. Training computes loss with a labels tensor that masks pad positions to -100, so they generate no gradient regardless of position, which makes right padding harmless.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Tokenizing with `padding=True` and then forgetting to pass `attention_mask` to the model. The model attends to the pad ids as if they were real text and quality drops.

60 second bullets to scan on the way to the call.

Define the pad token and explain why batching forces it into existence.
Describe how the attention mask makes pad positions inert inside the softmax.

Primary sources. Browse if you want the original framing.