Train loss drops while eval loss climbs at step 1200: name it and the standard fixes

Short answer·Medium·4.0 · 0·~3 min·Relevant atDatabricks Meta OpenAI

Attempt it

During an SFT run, train loss keeps dropping smoothly but eval loss bottoms out around step 1200 and starts climbing while train loss continues down. Name what this pattern is and walk through the two standard responses an engineer should take.

Free · 2 AI evals / day

**This is overfitting.** Train loss dropping while eval loss climbs is the textbook signature, and seeing it explicitly in your logs is one of the cleaner signals fine-tuning gives you. The model is increasingly memorizing the training set rather than learning generalizable structure. It scores better and better on the exact examples it has seen and worse and worse on held-out examples it has not. At step 1200 in this scenario, that handoff point has clearly passed. Two standard responses, usually combined. First, **early stopping**. Ship the checkpoint at the eval-loss minimum, in this case around step 1200, not the final-step checkpoint. Hugging Face Trainer and most other frameworks support load_best_model_at_end on eval-loss for exactly this reason. Set evaluation_strategy='steps' with a reasonable eval interval and save the best checkpoint automatically. Second, **reduce capacity or increase regularization** for the next run. For a LoRA fine-tune, the concrete levers are: lower the rank (r=32 to r=16, or r=16 to r=8), add or increase LoRA dropout (typically 0.05 to 0.1), drop the learning rate, or shorten the schedule. Each of these makes it harder for the model to memorize. A useful diagnostic before reaching for fixes: confirm the eval set is truly held out and not leaked from training. A leaky split sometimes hides this signal until something shifts. If the dataset is small (under a few thousand examples), more data is usually the deeper fix.

**Overfitting.** The divergence between train loss (still dropping) and eval loss (now rising) is the textbook signature. The model is increasingly memorizing training examples rather than learning generalizable structure. Two standard responses, often combined. **Early stopping.** Ship the checkpoint at the eval-loss minimum, around step 1200 here. Past that point, additional training degrades the model on data it will actually see in production. Hugging Face Trainer supports load_best_model_at_end with metric_for_best_model='eval_loss' to handle this automatically. Set evaluation_strategy to 'steps' with a reasonable interval (every 50 to 200 steps) and save_total_limit to keep memory bounded. The selected checkpoint is the one to deploy. **Reduce effective capacity or increase regularization** for the next run. For a LoRA fine-tune, the concrete levers in roughly descending order of effect: lower the rank (r=32 to r=16 to r=8), add or increase LoRA dropout (typical range 0.05 to 0.1), drop the learning rate (often 2x to 4x), shorten the training schedule. Generic ML levers like weight decay help less in LoRA settings because the constraint is the low-rank adapter shape, not weight magnitude. If the dataset is small (under a few thousand examples), the deeper fix is more data. Overfitting at step 1200 on a tiny dataset is the model running out of signal, not a hyperparameter mistake. Synthetic augmentation, data mixing with adjacent tasks, or active labeling are all options. Diagnostic before reaching for fixes: verify the eval set is genuinely held out. A leaky split often masquerades as healthy generalization until something shifts, and the cleanest tell is suspiciously close train and eval curves that abruptly diverge.

Quick reference

TL;DR

This is overfitting. Two responses: ship the checkpoint at the eval-loss minimum (early stopping), and reduce effective capacity or add regularization for the next run.

Memory aid

Easy to grasp

Imagine a student cramming for an exam by memorizing every practice question word for word. They get faster and faster on the practice set, but when the real exam shows even slightly different questions, they start failing because they never learned the underlying ideas. That gap between practice scores rising and exam scores falling is the signal that memorization has taken over from real learning. Two fixes work. First, stop them cramming the moment their practice and exam scores diverge, and use the version of them from that moment, not the over-crammed one. Second, give them fewer practice questions to obsess over, or harder questions that punish memorization, so they have to learn the ideas instead.

Key concepts

Overfitting Evaluation Early stopping Regularization

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: naming the overfit pattern, early stopping mechanics with load_best_model_at_end, the next-run capacity and regularization levers, when more data is the deeper fix, and the leakage diagnostic to rule out first.

Side-by-side

Symptom	Likely cause	First action
Train down, eval up (this case)	Overfitting	Early stop at eval min; lower capacity next run
Train flat, eval flat	Optimization broken (zero LR, masked labels)	Diagnose gradient path before changing capacity
Train down, eval flat	Eval set too easy or leaked	Verify eval set is held out and harder than train
Train down then spikes, eval also spikes	Numerical instability (fp16 overflow)	Switch to bf16, add gradient clipping
Train down, eval down (healthy)	Healthy generalization	Continue training to schedule end

Real-world examples

Real products, models, and research that use this idea.

Hugging Face Trainer's load_best_model_at_end plus metric_for_best_model='eval_loss' is the standard production fix for early stopping at the eval minimum.
Axolotl YAML configs expose save_strategy and evaluation_strategy directly so teams can pin the best checkpoint without code changes during 2026 Llama 4 fine-tunes.
Unsloth single-GPU recipes for Qwen 3.5 routinely default to LoRA dropout 0.05 to dampen memorization on small instruction sets.
TRL's SFTTrainer and DPOTrainer both inherit Hugging Face Trainer's eval-checkpoint selection, so the same load_best_model_at_end pattern works across SFT and preference tuning.
Open-source Mistral Large 3 community fine-tunes typically run for a single epoch on curated SFT sets to avoid hitting the overfit knee in the first place.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat if early stopping picks a checkpoint that is too early and underfits?

QHow do you choose between lowering rank versus adding LoRA dropout?

QWhy might weight decay help less in LoRA settings than in full fine-tuning?

QHow would you adjust your eval cadence to catch the inflection point more precisely?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Shipping the final-step checkpoint because the training loss looks best. The model that overfit hardest is not the model you want in production; the eval-minimum checkpoint is.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.