Training loss draws a sharp saw tooth at every epoch boundary: diagnose.

Short answer·Medium·4.0 · 0·~3 min·Asked atMckinsey Moveworks OpenAI·Relevant atDatabricks Meta

Attempt it

An SFT run shows a striking pattern: loss drops smoothly within each epoch, then jumps back UP at the moment a new epoch starts, producing a sharp saw tooth aligned exactly with epoch boundaries. Training is otherwise stable. Identify the most likely cause, why the jumps happen specifically at epoch boundaries, and what one line fix usually resolves it. Also name the secondary suspect to rule out.

Free · 2 AI evals / day

**The pattern of a saw-tooth aligned exactly with epoch boundaries** is almost always a data-order problem, not a training-instability one. The most likely cause is that the data loader is not shuffling between epochs, or the shuffle seed is fixed across epochs. Each epoch presents batches in the same order. Within an epoch the optimiser adapts to that specific sequence and loss falls smoothly. When the next epoch starts, the loader hands back batch one again, which is harder than the last batch of the previous epoch, so loss jumps up. The jumps happen at epoch boundaries because the boundary is the reset of the iteration order. The model is still learning. Within-epoch loss is still falling, often slightly faster each epoch as the model gets better. The saw-tooth is a bookkeeping artifact of measuring loss on a non-stationary sample order. The one-line fix is to enable per-epoch shuffling with an epoch-dependent seed. In PyTorch, set shuffle equal to True in the DataLoader. In distributed training with a DistributedSampler, you also need to call sampler.set_epoch(epoch) at the start of each epoch, otherwise the sampler reuses the same shuffle every epoch. The secondary suspect to rule out is a curriculum or sort by length sampler that re-sorts at the top of every epoch. That gives the same visual pattern. Cosine restarts in the learning-rate scheduler usually produce drops at the boundary, not jumps up.

**The diagnosis** is a data-order artifact, almost always missing per-epoch shuffling or a fixed shuffle seed. Each epoch presents batches in exactly the same order, so the optimiser smooths the loss through the sequence within an epoch, and when the next epoch starts the loader hands back batch one again. Batch one is harder than the final batch of the previous epoch (the model was tuned to those last examples just before the boundary), so loss jumps up. The saw-tooth aligns precisely with epoch boundaries because the boundary is the reset of the iteration order. **Why it is a measurement issue, not a training issue.** Within each epoch the curve still falls. Successive epochs typically reach lower per-batch loss values than previous epochs at the same iteration position, which is the signal that the model is still learning. The saw-tooth is the optimiser passing through the same batch sequence again on a slightly better model, not a sign of instability. **The fix.** Enable per-epoch shuffling with an epoch-dependent seed. In PyTorch's DataLoader, shuffle equal to True with the default sampler does the right thing for single-GPU training. In distributed training the DataLoader uses DistributedSampler, and you must call sampler.set_epoch(epoch) at the top of each epoch loop, otherwise the sampler reuses the same shuffle every epoch and the saw-tooth persists even with shuffle equal to True. This is the one-line fix that teams forget under FSDP or DDP. **Secondary suspect to rule out.** A curriculum or sort by length sampler that re-sorts at the top of every epoch produces the same visual pattern. Check the sampler config. Cosine-restart learning-rate schedulers usually produce drops at the boundary, not jumps up, so the direction of the discontinuity tells you which suspect to chase first.

TL;DR

The data loader repeats the same batch order each epoch, so loss restarts at the same hard-batch sequence. Fix: per-epoch shuffling with an epoch-dependent seed, plus sampler.set_epoch under DDP or FSDP.

Memory aid

Easy to grasp

Picture practising a piano programme of twenty pieces in the same fixed order every day. Within a day you warm up on the first piece, get smoother by the tenth, and play the last few beautifully. Tomorrow you start again on piece one, which is hard from cold, and your score for that piece looks worse than where you ended yesterday. The score chart looks like a saw, jumping up every morning. The fix is to shuffle the order each day so you do not always restart on the same cold piece. In a training run, the same trick is to reshuffle the data on every full pass so the first chunks are not always the same hard ones.

Key concepts

Training debugging Data loader Data shuffling Loss curve

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

A saw-tooth loss curve aligned exactly with epoch boundaries is one of the most frequently misdiagnosed patterns in fine-tuning. It looks alarming on a training dashboard, the loss jumps up at every epoch start, looks like instability, and the natural first reaction is to lower the learning rate or check gradient clipping. Both responses slow training without fixing anything, because the cause is not in the optimiser at all. It is in the data loader.

The diagnostic is structural: the jumps align precisely with epoch boundaries, not with random steps, and they repeat with the same shape across epochs. That precision is the tell. Only one class of cause can produce a curve whose discontinuities align exactly with the iteration-counter resets that define epochs, and that class is data-order artifacts. The loader is presenting the same batches in the same order each epoch, and the loss reflects that repetition.

This deep dive walks the mechanism in detail, explains why per-epoch shuffling fixes it, covers the DistributedSampler gotcha that catches most multi-GPU runs, and lists the secondary suspects (curriculum samplers, cosine-restart schedules) whose visual signatures overlap. The audience is engineers debugging a real training run, not theorists, the goal is to make the correct diagnosis fast and apply the one-line fix without disturbing the rest of the recipe.

Why the same batch order produces the same loss curve

An SFT training run sees a sequence of batches and updates the model parameters after each one. Within an epoch the loss curve is a function of two things: the model's current parameters and the specific sequence of batches it sees. If you fix both inputs, the curve is deterministic up to numerical noise.

Now suppose the loader repeats the same batch order each epoch. At iteration zero of epoch one, the model sees batch A1 with its initial parameters and computes some loss L1A. By the end of epoch one, the parameters have moved and the final batch produces a much lower loss.

At iteration zero of epoch two, the loader hands back batch A1 again. The parameters have moved, so the loss on A1 is now lower than L1A, but it is higher than the loss on the final batch of epoch one was, because the model was specifically tuned on that final batch right before the boundary and is now back to batch A1. The jump from the last batch of epoch one to the first batch of epoch two is the saw-tooth.

The shape repeats across epochs because the batch sequence repeats. Each epoch's curve is roughly the same shape, shifted lower because the model has improved overall. The boundary between any two epochs produces the same kind of jump because it is always the transition from the last (well-fitted) batch back to the first (less-fitted) batch.

It is critical to recognise that the model IS learning during all of this. The within-epoch curve falls, epoch two reaches lower loss values than epoch one at comparable iteration positions, and the long-run trend is downward. The saw-tooth is purely a function of measuring loss on a non-stationary sample order.

Why per-epoch shuffling fixes it

The DistributedSampler gotcha

Secondary suspects that look similar

Why the model is still learning, and what to do about it

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

PyTorch's DistributedSampler documentation explicitly warns that set_epoch must be called or shuffling will repeat, a warning many teams discover only after seeing this curve.
Hugging Face Trainer handles set_epoch automatically since transformers 4.x, but custom training loops with raw PyTorch DDP often miss it.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you distinguish the data-order saw-tooth from a cosine-restart artifact if both happen to align with epoch boundaries?

Look at the direction. Data-order produces jumps UP at the boundary, because the new epoch starts on the hardest batch of the cycle. Cosine restart produces drops or initial spikes because the learning rate jumps back to its high value, briefly destabilising loss before it settles. Also check the LR-versus-step plot, restarts are visible there.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Chasing this as a training-instability bug. It is a data-order artifact, the model is still learning, the loss curve is just being measured on a non-stationary sample order. Look at the loader, not the optimiser.

60 second bullets to scan on the way to the call.

What a saw-tooth pattern aligned with epoch boundaries indicates
Why the same batch order across epochs produces the same loss curve shape

Primary sources. Browse if you want the original framing.