A team is SFT'ing a strong base instruct model into a code-assistant product. They have access to high-quality code-instruction data (CodeAlpaca, Magicoder, internal code-review pairs) and general chat data (Tulu, UltraChat). Specify the starting mixture ratio, the role each component plays, how to tune the ratio, and what fails at both extremes.
Free · 2 AI evals / day
**Start at roughly 80% code instruction data and 20% general chat data.** The two slices play different roles.
**The code slice** (CodeAlpaca, Magicoder, internal code-review pairs) is the primary signal. It carries the behaviour the product needs: writing code, explaining bugs, suggesting fixes.
**The general slice** (Tulu, UltraChat, OpenAssistant) is anti-forgetting replay. Its job is to refresh the broad-chat distribution the base already had, so the model does not lose its ability to explain code in plain English or hold a follow-up conversation.
**Interleave the two at the row level**, not as big blocks. If you concatenate code rows then chat rows, the model overfits the early block and partially forgets it. Per-batch mixing keeps both signals present every optimizer step.
**Tune empirically.** Run pilots at 90/10, 80/20, 70/30. Read a code metric (HumanEval, MBPP, or internal pass-at-1) and a general metric (MMLU, MT-Bench). Pick the lowest general share that keeps the general metric within 1 to 2 points of base.
**Both extremes fail in opposite ways.** Below 5% general, catastrophic forgetting: code climbs but general drops. Above 30% general, code plateaus because batches are mostly off-topic.
**Start at 80% code / 20% general**, with the two slices doing distinct jobs.
**The code slice is the primary signal.** It contains CodeAlpaca, Magicoder, and internal code-review pairs, the data that teaches product behaviour. Most of the gradient comes from here.
**The general slice is anti-forgetting replay.** It contains Tulu, UltraChat, OpenAssistant, broad-chat data that refreshes the base's general capabilities. It does not teach new behaviour; it prevents the base's existing capabilities from being washed out.
**Mix at the row level, not in blocks.** Per-batch interleaving keeps both signals present every optimizer step. Block-concatenation creates curriculum effects: the model overfits the early block and partially forgets it later.
**Tune empirically.** Run three pilots at 90/10, 80/20, 70/30. Read two evals in parallel: a code metric (HumanEval, MBPP, internal pass-at-1) and a general metric (MMLU, MT-Bench).
The right ratio is the lowest general share that keeps the regression within tolerance (under 1 to 2 points) while the code metric climbs.
**Both extremes fail in opposite ways.** Under 5% general gives catastrophic forgetting: HumanEval climbs, MMLU and MT-Bench drop, users complain the model forgot how to explain. Over 30% general plateaus the code metric because batches are mostly off-topic, and you ship a weaker assistant than the tighter mix would produce.
The 70 to 85 percent domain range is the sweet spot for most code SFT runs.
Quick reference
TL;DR
Start at 80% code / 20% general, with the general slice acting as anti-forgetting replay. Interleave at the row level. Tune empirically with pilots at 90/10, 80/20, 70/30 reading both a code metric and a general metric.
Picture training a specialist doctor who also has to talk to patients in plain language. Most of the training is medical cases, that is what makes them good at the speciality. But a small share is conversational practice, because if they spend a year on only technical cases they forget how to explain a diagnosis to a frightened parent. The medical slice teaches the new skill. The conversational slice keeps the old skill from rotting. The right balance is mostly medical with enough conversation to keep the bedside manner, and finding it means watching both their technical scores and their patient-feedback scores. Both extremes fail in opposite ways.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You are doing continued pretraining on a domain corpus, not instruction-tuning Continued pretraining has different dynamics; the replay slice typically needs to be larger and the mixing rules are different from SFT.
Your base model already speaks your domain fluently and you only need surface-level adjustment A light SFT run with mostly general data and a small domain slice may suffice; the 80/20 ratio is overkill for narrow tuning.
You are fine-tuning a model that will only ever be used in the domain and never for general chat The replay slice exists to preserve general capabilities; if you never need them, you can reduce or remove the slice, accepting forgetting as acceptable.
Quick scan · 2–4 min
Everything important, quickly.
3 min: name the 80/20 starting ratio, distinguish the two slice roles, describe the row-level interleaving and the pilot protocol, and end with both failure modes.
Real-world examples
Real products, models, and research that use this idea.
1DeepSeek V4 Coder, a 2026 code-specialist model, uses a high-domain SFT mix with documented anti-forgetting replay drawn from general chat sources.
2Allen AI's Tulu 3 mixture demonstrates the two-slice structure at open-data scale, with per-slice ablation results showing the trade-off the question describes.
3Anthropic's published guidance on building Claude Opus 4.7 character training discusses general-capability replay as a deliberate component of specialist tuning.
4Llama 4 Maverick fine-tunes for medical and legal verticals in 2026 use the same domain plus replay structure with domain shares typically in the 75 to 85 percent range.
5Hugging Face's TRL examples for code-specialist fine-tunes recommend per-batch row-level interleaving and provide the dataset mixing utilities needed to implement it.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWalk through how you would adapt the 80/20 mix for a fine-tune on a low-resource programming language the base model barely knows.
Sign in or go Pro to see the approach for this follow-up.
QHow would you measure whether the anti-forgetting replay is actually working, beyond just reading MMLU?
Sign in or go Pro to see the approach for this follow-up.
QWhy does row-level interleaving outperform block-concatenation even though both sees the same total tokens?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Going 100% domain and discovering at deployment that the model lost its ability to explain code in plain English or hold a multi-turn conversation. The anti-forgetting replay slice is small but load-bearing.