Gradient Accumulation

Fill the conditions that preserve accumulation to large batch equivalence

Find the schedule assumption that breaks when accumulation steps change

Step versus epoch in a training run: define the relationship.

Compute the effective batch from per device batch, accum steps, and GPU count

Micro batch vs effective batch in fine-tuning: give the relationship

When is gradient accumulation NOT equivalent to a real larger batch?