Compute the effective batch from per device batch, accum steps, and GPU count

Fill in blank·Easy·4.0 · 0·~1 min·Asked atAi21 Cognizant Fiddler Ai·Relevant atDatabricks

Attempt it

With per_device_train_batch_size=2, gradient_accumulation_steps=8, and 4 GPUs in data parallel, the effective batch the optimizer sees per update is examples. The arithmetic is per_device times times world_size.

The **effective batch size** is the number of examples whose gradients contribute to a single optimizer update. It is the number that determines training dynamics, not the per-device batch size. Three configuration knobs combine to produce it. **`per_device_train_batch_size`** is how many examples one GPU processes in a single forward and backward pass. **`gradient_accumulation_steps`** is how many of these per-device micro-batches are accumulated before the optimizer actually steps. **`world_size`** is the number of GPUs running data-parallel training. In data-parallel mode, every GPU runs its own micro-batches in parallel and the gradients are averaged across GPUs before each step. So a single optimizer step integrates: - `per_device_train_batch_size` examples per micro-batch - times `gradient_accumulation_steps` micro-batches per step - times `world_size` GPUs running in parallel For 2 examples per device, 8 accumulation steps, and 4 GPUs: 2 times 8 times 4 equals 64 examples per optimizer update. The practical reason this matters is that learning-rate scaling rules key off the effective batch, not the per-device batch. Doubling accumulation steps or adding GPUs without retuning the learning rate often produces unexpectedly fast or slow convergence.

**Effective batch = per_device_train_batch_size times gradient_accumulation_steps times world_size**. For the given configuration (2, 8, 4 GPUs) the effective batch per optimizer update is 64 examples. The arithmetic captures three orthogonal axes of batch construction. The per-device batch is the chunk one GPU processes in a single forward-backward pass, bounded by GPU memory. The accumulation factor is the number of these chunks summed before the optimizer steps, used to grow the effective batch without growing memory. The world size is the number of data-parallel ranks, all contributing gradients to each step via an all-reduce. The optimizer sees one combined gradient computed from all `64` examples, and applies one update based on that gradient. Mathematically this is identical to training on a single batch of 64 examples in a single forward pass, provided no layer in the network couples examples across the batch dimension (BatchNorm would; LayerNorm and RMSNorm do not, which is why the equivalence holds cleanly for transformer fine-tuning). The practical consequence is that the learning rate should be tuned for the effective batch size, not the per-device batch size. A common bug is to keep the LR fixed when changing accumulation steps or scaling out to more GPUs, then blame convergence issues on the optimizer when the real cause is a now-wrong learning rate. The linear LR scaling rule (LR proportional to batch size) and the square-root rule (LR proportional to sqrt of batch size) are the two starting points; pick one and tune from there.

TL;DR

Effective batch is per_device_train_batch_size times gradient_accumulation_steps times world_size. With 2, 8, and 4 GPUs that is 64 examples per optimizer update.

Memory aid

Easy to grasp

Imagine four bakery teams each baking trays of cookies. Each team's oven only fits two trays at a time, so they bake two trays per oven round. They keep collecting cookies into a shared pile and only deliver the pile to the storefront after eight oven rounds. That means each team contributes 2 trays per round times 8 rounds equals 16 trays before delivery. Four teams doing this in parallel deliver 4 times 16 equals 64 trays per delivery cycle. The optimizer is the storefront and one delivery cycle is one update. The accumulation steps are the oven rounds, and the GPUs are the bakery teams.

Key concepts

Gradient accumulation Effective batch size Data-parallel training

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Effective batch size is the most-misunderstood number in distributed fine-tuning. It sits at the intersection of three different configuration knobs, and missing any of them produces a number off by a factor that silently degrades convergence.

The formula itself is simple: multiply the per-device micro-batch by the number of accumulation steps and by the number of data-parallel ranks. The full effective batch is the number of training examples whose gradients are averaged together before each optimizer step, and it is the number that determines training dynamics.

This deep dive walks through the three axes that contribute, the worked example from the question, the practical implications for learning-rate tuning, and the subtle traps around loss reduction and variable-length sequences that can break the arithmetic in ways that are not obvious until you profile a misbehaving run.

The three axes that combine to produce effective batch

Effective batch is the product of three independent factors, each corresponding to a different way of distributing work.

Per-device batch size is the chunk one GPU processes in a single forward and backward pass. It is bounded by activation memory: more examples per pass means more activations cached for backward, which is the dominant memory term in transformer training. On an 80GB H100 with a 7B model at 2048 sequence length, the per-device batch typically tops out around 4 to 8 without gradient checkpointing.

Gradient accumulation steps is the number of these per-device passes that are summed in the gradient buffers before the optimizer steps. Each pass adds its gradient to the running sum in-place; no extra memory is needed beyond the gradient buffers themselves. After the configured number of passes, the optimizer steps once on the accumulated gradient and the buffers are zeroed. This is the memory versus throughput axis: doubling accumulation steps doubles the effective batch without increasing memory, at the cost of doubling the number of forward and backward passes per optimizer step.

World size is the number of data-parallel ranks. Each rank holds a complete copy of the model and runs its own forward-backward passes on its own slice of data. Gradients are averaged across ranks via an all-reduce before each optimizer step, so every rank contributes its examples to the effective batch. This is the spatial axis: more ranks means more examples processed in parallel, with linear scaling subject to interconnect bandwidth.

These three axes multiply because they are orthogonal. World size 4 with per-device batch 2 and accumulation steps 8 means 4 ranks each process 2 examples per pass and accumulate 8 passes before stepping, for a total of 64 examples per optimizer update.

Worked example: 2 times 8 times 4 equals 64

Learning rate scaling and why this number matters

Traps that break the simple arithmetic

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Hugging Face Trainer logs effective_train_batch_size at startup as per_device times accum times world for exactly this reason.
Axolotl's YAML configs expose all three knobs explicitly so users can compute the effective batch before launching long Llama 4 fine-tuning runs.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does tensor parallel not multiply the effective batch the way data parallel does?

Trace what each parallelism shards. Data parallel replicates the model and shards the batch, so each rank contributes to the effective batch; tensor parallel shards the model itself and replicates the batch, so the same batch is processed once across all ranks.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Forgetting to multiply by world_size when computing effective batch. The optimizer sees the combined gradient from all data-parallel ranks, not just one rank's contribution.

60 second bullets to scan on the way to the call.

The three-factor formula: per_device times accum times world_size
Why data parallel multiplies the effective batch but tensor parallel does not

Primary sources. Browse if you want the original framing.