You are fine-tuning Llama 3 8B on variable length instruction data. Which padding strategy and side do you pick?

MCQ·Medium·4.0 · 0·~1 min·Asked atMidjourney Sambanova Two Sigma·Relevant atMeta

Attempt it

Two decisions for padding: the **strategy** (how much to pad) and the **side** (which end gets the filler). Both have right answers for fine tuning Llama 3 8B on variable length instruction data. **Strategy.** Use `padding='longest'`, which pads each batch only up to the longest sequence in that batch. Compare against `padding='max_length'`, which pads every batch to the model's full `model_max_length`. If your data has a wide length distribution (a few long examples mixed with many short ones), `max_length` wastes most of the GPU's compute on pad positions, which is 50 to 90 percent in typical instruction datasets. To recover more, add sequence bucketing (group examples of similar length together into batches) or sequence packing (concatenate multiple short examples into one row with a block diagonal attention mask). **Side.** Inference requires left padding for a decoder only model because generation starts from the last position in the sequence. The last token has to be a real one; right padding puts pad there and the model generates from a filler slot. For training, either side works as long as the loss is masked: set `labels = -100` at pad positions so they contribute no gradient. **Always pass the attention mask.** Without it the model attends to pad as if it were content, and even though the model still produces fluent output, quality drops in a way that does not throw any error.

For decoder only Llama 3 8B on variable length instruction data, the right answer is **`padding='longest'` plus a bucketing or packing strategy, left padding at inference, either side at training, always with the attention mask**. **Strategy choice.** `padding='longest'` (pad to the longest in each batch) is the right default because pad tokens cost FLOPs even though they contribute nothing to the loss. `padding='max_length'` (pad every batch to `model_max_length`) is wrong for skewed data because most of the compute goes to filler positions. Sequence bucketing (group same length examples into batches) cuts waste further. Sequence packing (concatenate short examples into one row with a block diagonal mask so each example only attends within itself) is the most efficient option and is supported by Flash Attention 2 and 3. **Padding side.** Inference must be left padded because a decoder only model generates from the last position; the last index of the sequence has to be a real token, and right padding puts pad there. Training works with either side as long as the loss is masked. The convention in many fine tuning recipes is right padding at training (more cache friendly) and left padding at inference (correctness requirement). **Attention mask is non negotiable.** Without it the model attends to pad as if it were real content and quality drops without an obvious symptom. The mask is also what makes pad inert at the loss level: combined with `labels = -100` at pad positions, pad contributes no gradient. **Why the distractors fail.** `max_length` to `model_max_length` wastes most of the compute. Batch size 1 throws away the parallelism the GPU exists for. Trusting the model to ignore pad ids without the mask is a popular wrong answer; the model does not.

TL;DR

Left-pad at inference (decoder only generates from the last position), either side at training with the attention mask. Use `padding='longest'` plus bucketing or packing, not fixed `max_length`.

Memory aid

Easy to grasp

Imagine a row of bookshelves where every shelf has to be the same width. You can either jam every shelf to the warehouse's maximum width (mostly air), or you can size each shelf to the widest book in that group (much less air). The second way wastes far less wood. That is `padding='longest'` versus `padding='max_length'`. Then there is the question of which side of the shelf to put the air pocket on. For a model that reads a story and continues writing it from the last word, the last book on the shelf has to be a real one, not air. So air goes on the left side. That is left padding for decoder only inference.

Key concepts

Padding Padding side Sequence packing

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Padding decisions look like implementation detail until they double your training cost or quietly degrade your inference quality. For fine tuning a decoder only Llama 3 8B on variable length instruction data, the right combination is padding='longest' plus bucketing or packing for compute efficiency, left padding at inference for correctness, either side at training as long as the attention mask is set and pad labels are masked to -100.

This explanation works through each decision separately, explains why the other options in the question fail, and closes on the two production bugs that come up most often when these decisions are made carelessly.

Strategy: longest, bucket, or pack, never max_length to ceiling

The strategy controls how much pad gets added to each batch. padding='longest' pads to the longest sequence in the current batch, which is the right default for most training. padding='max_length' pads to a fixed value (often the model's model_max_length), which is the wrong default for almost any real dataset.

The reason is the length distribution. Instruction tuning data is heavy tailed: most examples are a few hundred tokens, a small fraction are a few thousand. If you pad every batch to 8k (Llama 3's base context), 80 to 90 percent of the per step FLOPs go to pad positions that contribute no loss and produce no gradient. The model trains exactly as well on the unpadded portion but the per step cost is 5 to 10x what it needs to be.

Sequence bucketing reduces this further by grouping examples of similar length into the same batch. Within a bucket the pad waste is small. The downside is a small loss of randomization, which most teams accept.

Sequence packing is the most efficient option. Multiple short examples are concatenated into one row up to a target length (often the model's full context window), with a block diagonal attention mask so each example only attends within itself. Cross example attention is masked out. Flash Attention 2 and 3 expose varlen kernels that take per sequence offsets and run packed batches at near peak throughput. Axolotl, TRL, and Unsloth all default to packing for Llama 3 fine tuning in 2026.

Side: left at inference, either at training

The attention mask plays two roles

Why the distractors fail and the bug patterns to watch for

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Axolotl, TRL, and Unsloth all default to sequence packing for Llama 3 instruction fine tuning in 2026, with `sample_packing=true` in the config.
Hugging Face `Trainer` with `DataCollatorForLanguageModeling` defaults to right padding for training and handles the loss mask via `labels = -100` automatically.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is sequence packing more efficient than longest padding plus bucketing?

Even bucketed batches have within bucket length variance, so there is still some pad waste. Packing concatenates short examples to fill each row up to a fixed length, dropping waste to near zero. The cost is a block diagonal attention mask that Flash Attention 2 and 3 handle efficiently. Loss is still masked at the seams so examples do not contaminate each other.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Right padding a Llama 3 model at inference and watching the model generate from a pad position because the last index of the sequence is pad, not the user's last real token.

60 second bullets to scan on the way to the call.

State why a decoder only model requires left padding at inference
Contrast padding='longest' against padding='max_length' in compute terms

Primary sources. Browse if you want the original framing.