The correct choices are **A, B, and E**.
**A (task diversity)**: covering many distinct skills, summarisation, classification, code, reasoning, extraction, translation, is the single strongest predictor of post fine-tuning general capability. A 1,000-example set that touches 50 task types consistently beats a 10,000-example set that touches only 5.
**B (output-format diversity)**: models that have practised producing JSON, code blocks, lists, prose, and tables during SFT remain fluent in all of them at inference. Models trained only on one output shape regress when asked to produce others.
**E (prompt-template diversity)**: varying how the same task is framed (different phrasings, system prompts, instruction structures) teaches the model that the task identity lives in the request semantics, not in a single surface form.
The other options do not belong. **C (token-count diversity)** is fine to have for balanced batching but is a hygiene concern, not a quality driver. **D (tokenizer-version diversity)** is anti-recommended; tokenizer drift is a bug to pin down, not a regularisation strategy. **F (hardware diversity)** is irrelevant; the trained weights do not encode information about which GPU produced them.
The underlying principle is the LIMA / Tulu line: past a quality floor of a few thousand carefully curated examples, additional gains come from breadth on the dimensions that matter, not from raw volume on dimensions that do not.
**The right choices are A, B, and E**, the three axes that align with the LIMA / Tulu line on instruction tuning.
**Option A is task diversity.** Coverage across distinct skill categories is the strongest single predictor of post-FT general capability. A curated 1,000-example set covering 50 task types reliably outperforms a 10,000-example set covering 5. Each new task type adds a dimension to the manifold the model has to learn, and breadth transfers better than depth on any single slice.
**Option B is output-format diversity.** Models that practised JSON, code, prose, lists, and tables during SFT remain fluent in all of them at inference. Models trained on a single shape regress measurably when asked to produce others. The forgetting is not absolute but is sharp enough to matter for production.
**Option E is prompt-template diversity.** Varying how the same task is expressed under several phrasings and system-prompt styles teaches the model that task identity lives in the request semantics, not in a surface pattern. A model trained on one template locks in to it and breaks when users phrase things differently.
The three rejected options each fail for a different reason. **C (token-count diversity)** is a batching hygiene concern; wide length distributions complicate efficient packing but are not a quality driver. **D (tokenizer-version diversity)** is anti-recommended; tokenizer changes are bugs to eliminate, not regularisation, and mixing versions produces inconsistent special-token handling. **F (hardware diversity)** is irrelevant; numerical differences between GPUs average out and the trained weights do not reflect which silicon produced them.
The meta-principle: past a modest quality threshold, the marginal value of more examples is small compared to better coverage on the breadth axes that shape what the model can do.
Quick reference
TL;DR
A, B, E are the real diversity axes: task type, output format, and prompt template. C is hygiene, D is anti-recommended, F is unrelated to model quality.
Think of training a musician. Once they can play in tune, you do not make them better by giving them more copies of the same song. You make them better by giving them different songs, different instruments, and different settings to play in. Three of the answer choices line up with that idea: many kinds of tasks, many ways of presenting the answer, many ways of asking the question. The other three are like saying the musician should rotate music stands, switch sheet-music fonts, or play in different concert halls. None of those teach the musician anything useful. The same principle applies to instruction tuning. Past a quality floor, breadth on the things that matter beats volume on the things that do not.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Very small datasets below the LIMA quality threshold of a few hundred examples At very low volume, raw example count is still the binding constraint and breadth-first curation is premature.
Single-task narrow fine-tunes where preserving general capability is not a requirement If the deployment context only needs one task and one format, the diversity axes do not apply and depth on the single task is the right strategy.
Distillation runs where the student is being taught to mimic one teacher's exact output style Format and template diversity work against the mimicry objective; the distillation goal is single-style reproduction, not breadth.
Quick scan · 2–4 min
Everything important, quickly.
6 min: pick A, B, E, justify each as a real breadth axis that shapes model behaviour, then walk through why C is hygiene, D is anti-recommended, and F is unrelated to model quality.
Real-world examples
Real products, models, and research that use this idea.
1The LIMA paper from Meta demonstrated competitive instruction tuning with 1,000 carefully curated examples, establishing the diversity-over-volume principle.
2Tulu 3 from Allen Institute follows the diversity-first approach with task, format, and template coverage as explicit curation criteria.
3OpenAI's instruction-following recipe and Anthropic's HHH datasets both prioritise breadth on task and template axes over raw example count.
4Modern flagship instruction tuning for Claude Opus 4.7 and Llama 4 explicitly enumerate diversity axes during data curation rather than treating them as emergent properties.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you measure coverage on the task and format axes during data curation?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the relationship between Tulu-style diversity and the synthetic data generation approach?
Sign in or go Pro to see the approach for this follow-up.
QHow does prompt-template diversity interact with chat-template enforcement at inference?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating dataset size as the binding constraint. Past a quality threshold, breadth across task, format, and template axes matters far more than raw example count.