What is the sensible num_train_epochs for a typical instruction SFT run on a curated dataset?
1 to 3 epochs is the standard SFT range; the base already knows language, so a few passes adjust format and style without overfitting the small curated set.
Think of an expert chef being taught to plate a specific new dish. They already know how to cook, hold a knife, and balance flavors from decades of experience. The new lesson is just about plating style: the specific arrangement, the garnish, the sauce drizzle. You show them the technique once or twice, and they pick it up. If you make them practice the exact same plating fifty times in a row, they start producing identical-looking plates every time. This happens even for dishes that should look different. They have stopped learning the principle and started copying the example. SFT works the same way. The model already knows English. One to three passes over the curated examples teach the format. Many more passes turn the model into a parrot.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Epoch count is one of those hyperparameters where the published consensus is unusually clear. Of the four ranges (fractional passes, the standard 1 to 3, dozens, and fifty plus), three would produce models that no one would ship. One matches most published instruction tuning recipes of recent years.
The reasoning behind 1 to 3 epochs is structural. The pretrained base has already consumed trillions of tokens and learned language at a level that SFT cannot match. SFT's job is much smaller: nudge the output distribution toward instruction-following format and the desired response style. A small number of passes over a curated dataset is enough for that nudge. Many passes overfit and produce a parrot.
Why the standard range is 1 to 3 epochs
Three facts about SFT shape the epoch count regime.
Fact one: the base model has done the heavy lifting
Pretraining a 7B to 70B language model consumes trillions of tokens over weeks or months. The model emerges knowing English (and several other languages), basic syntax, world knowledge, simple arithmetic, code structure, and an enormous amount of fact-like content. The pretrained weights sit in a low-loss basin shaped by next-token prediction over an enormous corpus.
SFT does not need to teach any of that. The base already has it.
Fact two: SFT data is small and specific
A typical instruction tuning dataset is between 10K and 1M examples. Even at 1M examples with 1K tokens each, that is one billion tokens total, three to four orders of magnitude smaller than pretraining. The data is also curated and narrow: a few thousand instruction templates, a few response styles, a controlled vocabulary.
The goal is not to teach new content but to shift the model's output distribution toward this specific style of input and output.
Fact three: small data plus many passes equals memorisation
For a small curated dataset, each example is seen once per epoch. After a few epochs, the model has seen each example three or four times. That is enough to absorb the pattern without memorising the specific phrasings. After ten or twenty epochs, the model has seen each example ten or twenty times. That is enough to start memorising verbatim phrases and copying training set responses on related prompts.
Memorisation is the failure mode. The training loss continues to drop because the model is reciting the data. But held-out performance flatlines or drops, because the model has lost the diversity of phrasing that makes it useful on novel prompts.
The 1 to 3 epoch sweet spot
One to three epochs gives the model enough passes to absorb the instruction-following format and the desired response style. It does not give enough passes to memorise individual examples. This is the regime where the train versus held-out gap stays narrow. The model keeps its base capabilities while gaining the new format adjustment.
The exact number within the range depends on dataset size. Larger datasets need fewer epochs because each pass is more diverse. Smaller datasets benefit from a couple of extra passes to reinforce the format.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Hugging Face TRL's SFTConfig inherits num_train_epochs=3 from the Transformers TrainingArguments default, so many recipes start at 3 epochs unless they set it.
- Large instruction mixes of a million or more examples are often trained for a single epoch, because the diversity of the data already gives enough signal.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does dataset size affect the optimal epoch count and where is the rough crossover?
Small datasets (10K to 50K) benefit from 2 to 3 epochs for reinforcement. Large datasets (200K plus) often need only 1 epoch to avoid memorisation. There is no fixed crossover (Tulu 3 still used 2 epochs on about 939K examples), so let the held-out gap decide.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating SFT as if it were pretraining and running it for dozens of epochs. The base model already knows language. SFT only adjusts format and style, and over-training memorises the demonstrations instead of learning the underlying patterns.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.