During an SFT run, train loss keeps dropping smoothly but eval loss bottoms out around step 1200 and starts climbing while train loss continues down. Name what this pattern is and walk through the two standard responses an engineer should take.
Free · 2 AI evals / day
**This is overfitting.** Train loss dropping while eval loss climbs is the textbook signature, and seeing it explicitly in your logs is one of the cleaner signals fine-tuning gives you.
The model is increasingly memorizing the training set rather than learning generalizable structure. It scores better and better on the exact examples it has seen and worse and worse on held-out examples it has not. At step 1200 in this scenario, that handoff point has clearly passed.
Two standard responses, usually combined.
First, **early stopping**. Ship the checkpoint at the eval-loss minimum, in this case around step 1200, not the final-step checkpoint. Hugging Face Trainer and most other frameworks support load_best_model_at_end on eval-loss for exactly this reason. Set evaluation_strategy='steps' with a reasonable eval interval and save the best checkpoint automatically.
Second, **reduce capacity or increase regularization** for the next run. For a LoRA fine-tune, the concrete levers are: lower the rank (r=32 to r=16, or r=16 to r=8), add or increase LoRA dropout (typically 0.05 to 0.1), drop the learning rate, or shorten the schedule. Each of these makes it harder for the model to memorize.
A useful diagnostic before reaching for fixes: confirm the eval set is truly held out and not leaked from training. A leaky split sometimes hides this signal until something shifts.
If the dataset is small (under a few thousand examples), more data is usually the deeper fix.
**Overfitting.** The divergence between train loss (still dropping) and eval loss (now rising) is the textbook signature. The model is increasingly memorizing training examples rather than learning generalizable structure.
Two standard responses, often combined.
**Early stopping.** Ship the checkpoint at the eval-loss minimum, around step 1200 here. Past that point, additional training degrades the model on data it will actually see in production. Hugging Face Trainer supports load_best_model_at_end with metric_for_best_model='eval_loss' to handle this automatically. Set evaluation_strategy to 'steps' with a reasonable interval (every 50 to 200 steps) and save_total_limit to keep memory bounded. The selected checkpoint is the one to deploy.
**Reduce effective capacity or increase regularization** for the next run. For a LoRA fine-tune, the concrete levers in roughly descending order of effect: lower the rank (r=32 to r=16 to r=8), add or increase LoRA dropout (typical range 0.05 to 0.1), drop the learning rate (often 2x to 4x), shorten the training schedule. Generic ML levers like weight decay help less in LoRA settings because the constraint is the low-rank adapter shape, not weight magnitude.
If the dataset is small (under a few thousand examples), the deeper fix is more data. Overfitting at step 1200 on a tiny dataset is the model running out of signal, not a hyperparameter mistake. Synthetic augmentation, data mixing with adjacent tasks, or active labeling are all options.
Diagnostic before reaching for fixes: verify the eval set is genuinely held out. A leaky split often masquerades as healthy generalization until something shifts, and the cleanest tell is suspiciously close train and eval curves that abruptly diverge.
Quick reference
TL;DR
This is overfitting. Two responses: ship the checkpoint at the eval-loss minimum (early stopping), and reduce effective capacity or add regularization for the next run.
Imagine a student cramming for an exam by memorizing every practice question word for word. They get faster and faster on the practice set, but when the real exam shows even slightly different questions, they start failing because they never learned the underlying ideas. That gap between practice scores rising and exam scores falling is the signal that memorization has taken over from real learning. Two fixes work. First, stop them cramming the moment their practice and exam scores diverge, and use the version of them from that moment, not the over-crammed one. Second, give them fewer practice questions to obsess over, or harder questions that punish memorization, so they have to learn the ideas instead.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Eval set is leaked from train, so the overfit signal is the leak surfacing Capacity reduction does not fix data leakage; the right move is to re-split and re-evaluate.
Train and eval curves are both rising or unstable This is not overfitting; it is an optimization problem (numerical, schedule, or data quality).
Eval set is too small for noise to be distinguished from signal Eval-loss minima with very small eval sets can swing on a few examples; expand the eval set before tuning on it.
Quick scan · 2–4 min
Everything important, quickly.
4 min: naming the overfit pattern, early stopping mechanics with load_best_model_at_end, the next-run capacity and regularization levers, when more data is the deeper fix, and the leakage diagnostic to rule out first.
Side-by-side
Symptom
Likely cause
First action
Train down, eval up (this case)
Overfitting
Early stop at eval min; lower capacity next run
Train flat, eval flat
Optimization broken (zero LR, masked labels)
Diagnose gradient path before changing capacity
Train down, eval flat
Eval set too easy or leaked
Verify eval set is held out and harder than train
Train down then spikes, eval also spikes
Numerical instability (fp16 overflow)
Switch to bf16, add gradient clipping
Train down, eval down (healthy)
Healthy generalization
Continue training to schedule end
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face Trainer's load_best_model_at_end plus metric_for_best_model='eval_loss' is the standard production fix for early stopping at the eval minimum.
2Axolotl YAML configs expose save_strategy and evaluation_strategy directly so teams can pin the best checkpoint without code changes during 2026 Llama 4 fine-tunes.
3Unsloth single-GPU recipes for Qwen 3.5 routinely default to LoRA dropout 0.05 to dampen memorization on small instruction sets.
4TRL's SFTTrainer and DPOTrainer both inherit Hugging Face Trainer's eval-checkpoint selection, so the same load_best_model_at_end pattern works across SFT and preference tuning.
5Open-source Mistral Large 3 community fine-tunes typically run for a single epoch on curated SFT sets to avoid hitting the overfit knee in the first place.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat if early stopping picks a checkpoint that is too early and underfits?
Sign in or go Pro to see the approach for this follow-up.
QHow do you choose between lowering rank versus adding LoRA dropout?
Sign in or go Pro to see the approach for this follow-up.
QWhy might weight decay help less in LoRA settings than in full fine-tuning?
Sign in or go Pro to see the approach for this follow-up.
QHow would you adjust your eval cadence to catch the inflection point more precisely?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Shipping the final-step checkpoint because the training loss looks best. The model that overfit hardest is not the model you want in production; the eval-minimum checkpoint is.