Compute the effective batch from per-device batch, accum steps, and GPU count

Fill in blank·Easy·4.0 · 0·~1 min

Attempt it

With per_device_train_batch_size=2, gradient_accumulation_steps=8, and 4 GPUs in data-parallel, the effective batch the optimizer sees per update is examples. The arithmetic is per_device times times world_size.

The **effective batch size** is the number of examples whose gradients contribute to a single optimizer update. It is the number that determines training dynamics, not the per-device batch size. Three configuration knobs combine to produce it. **`per_device_train_batch_size`** is how many examples one GPU processes in a single forward and backward pass. **`gradient_accumulation_steps`** is how many of these per-device micro-batches are accumulated before the optimizer actually steps. **`world_size`** is the number of GPUs running data-parallel training. In data-parallel mode, every GPU runs its own micro-batches in parallel and the gradients are averaged across GPUs before each step. So a single optimizer step integrates: - `per_device_train_batch_size` examples per micro-batch - times `gradient_accumulation_steps` micro-batches per step - times `world_size` GPUs running in parallel For 2 examples per device, 8 accumulation steps, and 4 GPUs: 2 times 8 times 4 equals 64 examples per optimizer update. The practical reason this matters is that learning-rate scaling rules key off the effective batch, not the per-device batch. Doubling accumulation steps or adding GPUs without retuning the learning rate often produces unexpectedly fast or slow convergence.

**Effective batch = per_device_train_batch_size times gradient_accumulation_steps times world_size**. For the given configuration (2, 8, 4 GPUs) the effective batch per optimizer update is 64 examples. The arithmetic captures three orthogonal axes of batch construction. The per-device batch is the chunk one GPU processes in a single forward-backward pass, bounded by GPU memory. The accumulation factor is the number of these chunks summed before the optimizer steps, used to grow the effective batch without growing memory. The world size is the number of data-parallel ranks, all contributing gradients to each step via an all-reduce. The optimizer sees one combined gradient computed from all `64` examples, and applies one update based on that gradient. Mathematically this is identical to training on a single batch of 64 examples in a single forward pass, provided no layer in the network couples examples across the batch dimension (BatchNorm would; LayerNorm and RMSNorm do not, which is why the equivalence holds cleanly for transformer fine-tuning). The practical consequence is that the learning rate should be tuned for the effective batch size, not the per-device batch size. A common bug is to keep the LR fixed when changing accumulation steps or scaling out to more GPUs, then blame convergence issues on the optimizer when the real cause is a now-wrong learning rate. The linear LR scaling rule (LR proportional to batch size) and the square-root rule (LR proportional to sqrt of batch size) are the two starting points; pick one and tune from there.

Quick reference

TL;DR

Effective batch is per_device_train_batch_size times gradient_accumulation_steps times world_size. With 2, 8, and 4 GPUs that is 64 examples per optimizer update.

Memory aid

Easy to grasp

Imagine four bakery teams each baking trays of cookies. Each team's oven only fits two trays at a time, so they bake two trays per oven round. They keep collecting cookies into a shared pile and only deliver the pile to the storefront after eight oven rounds. That means each team contributes 2 trays per round times 8 rounds equals 16 trays before delivery. Four teams doing this in parallel deliver 4 times 16 equals 64 trays per delivery cycle. The optimizer is the storefront and one delivery cycle is one update. The accumulation steps are the oven rounds, and the GPUs are the bakery teams.

Key concepts

Gradient accumulation Effective batch size Data-parallel training

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: state the three-factor formula, walk through the worked example, then cover the loss-reduction trap and the LR-scaling implication that catch most teams.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face Trainer logs effective_train_batch_size at startup as per_device times accum times world for exactly this reason.
Axolotl's YAML configs expose all three knobs explicitly so users can compute the effective batch before launching long Llama 4 fine-tuning runs.
Unsloth recipes for Qwen 3.5 on a single 24GB GPU use per_device=4, accum=8, world=1 to reach an effective batch of 32 without OOM.
DeepSpeed and Accelerate's launcher both expose train_micro_batch_size_per_gpu and gradient_accumulation_steps as separate first-class config keys.
Multi-node Llama 4 Maverick fine-tunes at large compute orgs routinely use per_device=1, accum=32, world=64 to reach effective batches of 2048.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does tensor parallel not multiply the effective batch the way data parallel does?

QHow should you adjust the learning rate when scaling from 4 GPUs to 16 GPUs with the same per-device batch and accumulation?

QWhat goes wrong if you use per-token mean reduction across micro-batches with very different sequence lengths?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Forgetting to multiply by world_size when computing effective batch. The optimizer sees the combined gradient from all data-parallel ranks, not just one rank's contribution.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.