Micro batch vs effective batch in fine-tuning: give the relationship

Attempt it

Micro batch is what physically fits in one GPU's memory in a single forward/backward pass. It is bounded by activation memory: bigger sequences and bigger models force smaller micro batches.

Effective batch is what the optimizer actually sees per step: the number of examples whose gradients are averaged before the Adam update fires. It is the product of three things:

effective_batch = micro_batch × grad_accumulation_steps × num_data_parallel_gpus

Gradient accumulation lets you simulate a large effective batch on small hardware by averaging several micro batches before stepping the optimizer. Data parallelism multiplies the count again by running independent micro batches on each replica and all-reducing the gradients.

This distinction matters for both stability and reproducibility: the LR schedule, weight decay, and warmup were tuned against effective batch, so a run that uses the right effective batch via accumulation behaves like the original recipe even on smaller GPUs.

Your answer

Quick reference

TL;DR

Micro batch is what fits in one GPU in one pass. Effective batch is what the optimizer sees per step: micro batch times gradient accumulation steps times data-parallel GPU count.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine cooking a stew that serves 64 people, but your largest pot only holds 4 servings. Two options. You can cook 16 small pots one after another and pour them all into one big stockpot before serving (accumulating small pots over time), or you can have 16 cooks each making one small pot in parallel and pooling them (cooking on many stoves at once). Either way the diners receive the same 64-serving stew. The size of one pot is the micro batch: hardware constraint. The size of what reaches the dinner table is the effective batch: what the recipe actually cared about. The recipe was tuned for 64 servings, and as long as you reach 64 by any combination of more pots over time or more cooks in parallel, the dish tastes right.

Key concepts

Batch size Gradient accumulation Data parallelism Training systems

Quick scan · 2–4 min

Everything important, quickly.

4 min: define micro batch versus effective batch + three-factor product formula + why accumulation is equivalent for transformers + data-parallel multiplier + recipe portability via effective batch + LR scaling caveats.

Side-by-side

Aspect	Micro batch	Effective batch
What it is	Examples per GPU per forward-backward pass	Examples whose gradients are averaged before optimizer step
Bound by	Activation memory on one GPU	Recipe choice; product of micro, accumulation, world size
What the LR is tuned against	No (hardware artifact)	Yes (statistical quantity)
Effect of doubling it	Doubles activation memory	Roughly doubles per-step gradient signal to noise
Common typical value	1 to 8 per H100 on 7B model	32 to 256 for LoRA fine-tunes

Real-world examples

Real products, models, and research that use this idea.

Hugging Face Trainer's per_device_train_batch_size, gradient_accumulation_steps, and world_size combine multiplicatively into effective batch and are the canonical knobs in modern fine-tuning scripts.
QLoRA recipes for Llama 3.1 8B routinely set micro batch 4 with accumulation 8 on one GPU to hit an effective batch of 32, matching the published learning rate.
DeepSpeed and FSDP both expose data-parallel world size as the multiplier that lifts effective batch above what a single GPU could hold.
Axolotl and LLaMA-Factory documentation explicitly call out the three-factor formula so users can scale runs between dev workstations and production clusters without retuning the LR.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does linear LR scaling break down at very large effective batches?

Sign in or go Pro to see the approach for this follow-up.

QHow should you change accumulation when moving from 1 GPU to 8 to preserve a recipe?

Sign in or go Pro to see the approach for this follow-up.

QWhat goes wrong if your loss sums over tokens and you accumulate?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.