A, B, E are the real diversity axes: task type, output format, and prompt template. C is hygiene, D is anti-recommended, F is unrelated to model quality.
Think of training a musician. Once they can play in tune, you do not make them better by giving them more copies of the same song. You make them better by giving them different songs, different instruments, and different settings to play in. Three of the answer choices line up with that idea: many kinds of tasks, many ways of presenting the answer, many ways of asking the question. The other three are like saying the musician should rotate music stands, switch sheet-music fonts, or play in different concert halls. None of those teach the musician anything useful. The same principle applies to instruction tuning. Past a quality floor, breadth on the things that matter beats volume on the things that do not.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The Tulu and LIMA lines of research on instruction tuning made an unintuitive claim that has held up under replication: past a modest quality threshold, more data is not the path to a better instruction-tuned model. The path is better coverage on a specific set of breadth axes. A curated 1,000-example set that spans 50 task types and 5 output formats and 10 prompt templates outperforms a 10,000-example set that covers a narrow slice along all three axes. The diversity over volume principle is the most important single insight from the post-LIMA literature on SFT.
This question puts six candidate diversity axes on the table and asks the candidate to pick the three that genuinely shape what the model learns. Three of the options are the real axes: task type, output format, and prompt template. Three are decoys that look like diversity axes but are actually operational concerns (token-count distribution), actively harmful (tokenizer-version mixing), or completely irrelevant (hardware rotation).
A candidate who picks A, B, and E demonstrates understanding of the underlying mechanism: diversity axes matter when they exercise different parts of the model's representation and force it to learn the general structure rather than overfitting to a narrow surface form. Candidates who pick C, D, or F demonstrate that they have confused axes of variation with axes that the model can actually learn from, which is the most common conceptual mistake in instruction-tuning curation.
Task diversity (A): the strongest single axis
Task diversity is the breadth axis with the largest empirical effect on post fine-tuning general capability. The LIMA paper established this and the result has been replicated across multiple follow-up studies. A small curated corpus that covers many distinct task types (summarisation, classification, code, reasoning, extraction, translation, conversation, creative writing) consistently lifts more general capability than a larger corpus that focuses on a narrower slice.
The underlying mechanism is that each task type exercises a different combination of the model's internal circuits. Summarisation engages compression and salience detection. Classification engages decision-boundary logic. Code generation engages syntactic structure and constraint satisfaction. Reasoning engages multi-step inference and intermediate-state tracking. Translation engages cross-distribution alignment. If a training corpus exercises only one of these, the rest of the network's capabilities drift under the supervision signal and the model regresses on tasks the training data did not cover.
The practical implication is that curation should explicitly enumerate task types and ensure coverage. A pragmatic approach is to define a fixed taxonomy of 20 to 50 task types, tag every training example with its primary type, compute coverage across the taxonomy, and target the lowest-covered types for additional curation. This is what Tulu 3, OpenAI's instruction-following recipes, and Anthropic's HHH datasets do under the hood; the diversity is intentional, not emergent.
The quality threshold below which volume still matters is real but modest. For corpora below a few hundred examples, raw example count is still the binding constraint and you do need more data before breadth becomes the dominant lever. Past roughly 1,000 to 2,000 examples, breadth on the task axis starts to outperform additional depth on a narrow slice.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- The LIMA paper from Meta demonstrated competitive instruction tuning with 1,000 carefully curated examples, establishing the diversity-over-volume principle.
- Tulu 3 from Allen Institute follows the diversity-first approach with task, format, and template coverage as explicit curation criteria.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you measure coverage on the task and format axes during data curation?
Tag each training example with a task category (from a fixed taxonomy of 20 to 50 types) and an output format (from a fixed list of 5 to 10 shapes). Compute coverage as the count of distinct values in each axis and the entropy across the corpus; flag low-entropy axes for additional curation.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating dataset size as the binding constraint. Past a quality threshold, breadth across task, format, and template axes matters far more than raw example count.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.