You are fine-tuning Llama 3 8B on variable-length instruction data. Which padding strategy and side do you pick?

MCQ·Medium·4.0 · 0·~1 min·Relevant atMeta

Attempt it

Two decisions for padding: the **strategy** (how much to pad) and the **side** (which end gets the filler). Both have right answers for fine tuning Llama 3 8B on variable length instruction data. **Strategy.** Use `padding='longest'`, which pads each batch only up to the longest sequence in that batch. Compare against `padding='max_length'`, which pads every batch to the model's full `model_max_length`. If your data has a wide length distribution (a few long examples mixed with many short ones), `max_length` wastes most of the GPU's compute on pad positions, which is 50 to 90 percent in typical instruction datasets. To recover more, add sequence bucketing (group examples of similar length together into batches) or sequence packing (concatenate multiple short examples into one row with a block diagonal attention mask). **Side.** Inference requires left padding for a decoder only model because generation starts from the last position in the sequence. The last token has to be a real one; right padding puts pad there and the model generates from a filler slot. For training, either side works as long as the loss is masked: set `labels = -100` at pad positions so they contribute no gradient. **Always pass the attention mask.** Without it the model attends to pad as if it were content, and even though the model still produces fluent output, quality drops in a way that does not throw any error.

For decoder only Llama 3 8B on variable length instruction data, the right answer is **`padding='longest'` plus a bucketing or packing strategy, left padding at inference, either side at training, always with the attention mask**. **Strategy choice.** `padding='longest'` (pad to the longest in each batch) is the right default because pad tokens cost FLOPs even though they contribute nothing to the loss. `padding='max_length'` (pad every batch to `model_max_length`) is wrong for skewed data because most of the compute goes to filler positions. Sequence bucketing (group same length examples into batches) cuts waste further. Sequence packing (concatenate short examples into one row with a block diagonal mask so each example only attends within itself) is the most efficient option and is supported by Flash Attention 2 and 3. **Padding side.** Inference must be left padded because a decoder only model generates from the last position; the last index of the sequence has to be a real token, and right padding puts pad there. Training works with either side as long as the loss is masked. The convention in many fine tuning recipes is right padding at training (more cache friendly) and left padding at inference (correctness requirement). **Attention mask is non negotiable.** Without it the model attends to pad as if it were real content and quality drops without an obvious symptom. The mask is also what makes pad inert at the loss level: combined with `labels = -100` at pad positions, pad contributes no gradient. **Why the distractors fail.** `max_length` to `model_max_length` wastes most of the compute. Batch size 1 throws away the parallelism the GPU exists for. Trusting the model to ignore pad ids without the mask is a popular wrong answer; the model does not.

Quick reference

TL;DR

Left-pad at inference (decoder only generates from the last position), either side at training with the attention mask. Use `padding='longest'` plus bucketing or packing, not fixed `max_length`.

Memory aid

Easy to grasp

Imagine a row of bookshelves where every shelf has to be the same width. You can either jam every shelf to the warehouse's maximum width (mostly air), or you can size each shelf to the widest book in that group (much less air). The second way wastes far less wood. That is `padding='longest'` versus `padding='max_length'`. Then there is the question of which side of the shelf to put the air pocket on. For a model that reads a story and continues writing it from the last word, the last book on the shelf has to be a real one, not air. So air goes on the left side. That is left padding for decoder only inference.

Key concepts

Padding Padding side Sequence packing

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: distinguish strategy (longest vs max_length vs packing) from side (left vs right), explain why decoder only inference requires left padding, explain how packing eliminates pad waste, name the two roles of the attention mask, and close on the two production bugs (wrong side, missing mask).

Real-world examples

Real products, models, and research that use this idea.

Axolotl, TRL, and Unsloth all default to sequence packing for Llama 3 instruction fine tuning in 2026, with `sample_packing=true` in the config.
Hugging Face `Trainer` with `DataCollatorForLanguageModeling` defaults to right padding for training and handles the loss mask via `labels = -100` automatically.
vLLM and SGLang serving engines avoid request padding entirely via continuous batching plus paged attention, building a custom mask per request.
Flash Attention 2 and 3 expose `varlen` kernels that take per sequence offsets so packed batches run at near peak throughput.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is sequence packing more efficient than longest padding plus bucketing?

QIf training works with right padding but inference requires left padding, how do most teams handle the mismatch in production code?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Right padding a Llama 3 model at inference and watching the model generate from a pad position because the last index of the sequence is pad, not the user's last real token.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.