Micro batch vs effective batch in fine-tuning: give the relationship

Flashcard·Easy·4.0 · 0·~30s·Asked atNykaa Ola Redis·Relevant atCoreweave Databricks Fireworks Ai Lambda Labs

Attempt it

Micro batch is what physically fits in one GPU's memory in a single forward/backward pass. It is bounded by activation memory: bigger sequences and bigger models force smaller micro batches.

Effective batch is what the optimizer actually sees per step: the number of examples whose gradients are averaged before the Adam update fires. It is the product of three things:

effective_batch = micro_batch × grad_accumulation_steps × num_data_parallel_gpus

Gradient accumulation lets you simulate a large effective batch on small hardware by averaging several micro batches before stepping the optimizer. Data parallelism multiplies the count again by running independent micro batches on each replica and all-reducing the gradients.

This distinction matters for both stability and reproducibility: the LR schedule, weight decay, and warmup were tuned against effective batch, so a run that uses the right effective batch via accumulation behaves like the original recipe even on smaller GPUs.

Your answer

**Micro batch and effective batch are two different things** that both go by the name 'batch' in training code. Confusing them is a common source of reproducibility bugs. Micro batch is what physically fits on one GPU in a single forward and backward pass. It is bounded by activation memory: bigger sequence lengths and bigger models force you to use smaller micro batches because each one consumes more memory. Effective batch is what the optimizer actually sees before stepping. It is the number of examples whose gradients are averaged together before the parameter update fires. The relationship is: `effective_batch = micro_batch * grad_accumulation_steps * num_data_parallel_gpus` Gradient accumulation lets you simulate a large effective batch on small hardware by running several micro batches in sequence, accumulating their gradients, and only then stepping the optimizer. Data parallelism multiplies the count again by running independent micro batches on each GPU and all-reducing the gradients before the step. The distinction matters because the learning rate schedule, weight decay, and warmup were tuned against the effective batch. A run that hits the right effective batch via accumulation behaves like the original recipe, even on smaller GPUs.

**Two distinct quantities both called 'batch':** the micro batch is a hardware concept; the effective batch is a statistical concept. Micro batch is the number of examples that fit on one GPU in one forward-backward pass. It is bounded primarily by activation memory, which grows with sequence length, model size, and number of layers. On a 24 GB GPU fine-tuning a 7B model at sequence length 2048, the micro batch is often 1 or 2; on an 80 GB H100 it might be 4 or 8. Effective batch is the number of examples whose gradients are averaged before the optimizer steps. It is the product of three factors: $$\text{effective\_batch} = \text{micro\_batch} \times \text{grad\_accumulation\_steps} \times \text{num\_data\_parallel\_gpus}$$ Gradient accumulation contributes the first multiplier beyond micro batch. The trainer runs N micro batches in sequence, accumulating gradients in the parameter buffers without stepping. The optimizer steps once after the Nth pass. This is mathematically identical to running one big batch of N times the micro batch size, for any model whose layers are per-example (LayerNorm, RMSNorm, attention) rather than per-batch (BatchNorm). Data parallelism contributes the second multiplier. K replicas each run independent micro batches, compute local gradients, and all-reduce them before the optimizer step. The result is mathematically equivalent to a single replica running K times the local micro batch. The distinction matters because the LR schedule, weight decay, and warmup were tuned against the effective batch. Hitting the right effective batch via any combination of accumulation and data parallelism gives you reproducible training even when the hardware shape changes.

TL;DR

Micro batch is what fits in one GPU in one pass. Effective batch is what the optimizer sees per step: micro batch times gradient accumulation steps times data-parallel GPU count.

Memory aid

Easy to grasp

Imagine cooking a stew that serves 64 people, but your largest pot only holds 4 servings. Two options. You can cook 16 small pots one after another and pour them all into one big stockpot before serving (accumulating small pots over time), or you can have 16 cooks each making one small pot in parallel and pooling them (cooking on many stoves at once). Either way the diners receive the same 64-serving stew. The size of one pot is the micro batch: hardware constraint. The size of what reaches the dinner table is the effective batch: what the recipe actually cared about. The recipe was tuned for 64 servings, and as long as you reach 64 by any combination of more pots over time or more cooks in parallel, the dish tastes right.

Key concepts

Batch size Gradient accumulation Data parallelism Training systems

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Almost every fine-tuning reproducibility bug eventually traces back to a confusion between micro batch and effective batch. The two are both called 'batch' in training code, but they answer completely different questions and one cannot be substituted for the other without breaking the recipe. Internalising the distinction once saves an enormous amount of debugging time.

Micro batch is a hardware concept. It is the number of examples that fit on one GPU in one forward and backward pass, bounded by activation memory. On a 24 GB consumer GPU running a 7B model at 2048 tokens, the micro batch is often 1 or 2. On an 80 GB H100 it might be 4 or 8. The micro batch number tells you nothing about how the optimizer behaves; it only tells you what your hardware can hold.

Effective batch is a statistical concept. It is the number of examples whose gradients are averaged together before the optimizer takes a step. The learning rate, the weight decay, the warmup schedule, all of the optimisation hyperparameters were tuned against this quantity. Two runs with the same effective batch but different micro batches will converge to similar results; two runs with the same micro batch but different effective batches will not.

The relationship is a simple product: effective batch equals micro batch times gradient accumulation steps times the number of data-parallel GPUs. Each factor contributes a different mechanism, and any combination of values that reaches the target effective batch will reproduce the recipe. This deep dive walks through each factor in detail, explains why accumulation is mathematically equivalent to a large batch for transformers, shows why data parallelism contributes a multiplier rather than independent runs, and lays out the practical implications for porting recipes across hardware shapes.

The two definitions and why both names exist

Micro batch is the number of examples processed in one forward-backward pass on one GPU. It is the quantity you set as per_device_train_batch_size in Hugging Face Trainer or as train_micro_batch_size_per_gpu in DeepSpeed configs. The bound on this number is activation memory, which scales with batch size, sequence length, hidden dimension, and number of layers.

Effective batch is the number of examples whose gradients are averaged before the optimizer steps. It is the quantity the recipe is implicitly tuned against, even when the recipe documentation calls it 'batch size' without qualifying which kind.

Both names exist because the two roles diverged historically. In the early deep-learning era, GPUs were small relative to models and a single physical batch was the only kind of batch in play. As models grew, that single batch became too large to fit on one GPU and the field developed gradient accumulation (one GPU, multiple sequential micro batches) and data parallelism (multiple GPUs, one micro batch each, gradients averaged). Both techniques produce an effective batch larger than the micro batch, and the term 'batch' became ambiguous unless qualified.

The practical consequence is that any modern training config has at least two numbers that affect the optimizer's view of batch size: per_device_train_batch_size and gradient_accumulation_steps. Add a multi-GPU world size and you have three. The recipe author tuned for the product of all three; reproducing the recipe means hitting that product, by any combination.

Gradient accumulation: the first multiplier

Data parallelism: the second multiplier

Why effective batch is the tuning quantity

Practical patterns and the traps that bite

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Aspect	Micro batch	Effective batch
What it is	Examples per GPU per forward-backward pass	Examples whose gradients are averaged before optimizer step
Bound by	Activation memory on one GPU	Recipe choice; product of micro, accumulation, world size
What the LR is tuned against	No (hardware artifact)	Yes (statistical quantity)
Effect of doubling it	Doubles activation memory	Roughly doubles per-step gradient signal to noise
Common typical value	1 to 8 per H100 on 7B model	32 to 256 for LoRA fine-tunes

Real products, models, and research that use this idea.

Hugging Face Trainer's per_device_train_batch_size, gradient_accumulation_steps, and world_size combine multiplicatively into effective batch and are the canonical knobs in modern fine-tuning scripts.
QLoRA recipes for Llama 3.1 8B routinely set micro batch 4 with accumulation 8 on one GPU to hit an effective batch of 32, matching the published learning rate.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does linear LR scaling break down at very large effective batches?

Linear scaling assumes the gradient noise scale matches the optimizer's tolerance. At very large effective batches the per-step gradient becomes too low-noise for SGD-style optimisers to explore well, and the optimum effective step size grows sub-linearly. Square-root scaling is a common empirical compromise; warmup also has to lengthen because the first few steps with a large effective batch can destabilise.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating micro batch as the value the LR schedule was tuned against. Recipes are written for effective batch; using the wrong one quietly mis-scales the learning rate.

60 second bullets to scan on the way to the call.

The two definitions: micro batch as memory constraint, effective batch as optimizer-step quantity
The product formula linking micro batch, accumulation steps, and data-parallel GPU count

Primary sources. Browse if you want the original framing.