Order these instruction datasets from highest to lowest quality per example
- 1LIMA (Meta, 2023): 1000 hand curated high quality examples covering many tasks
- 2Alpaca (Stanford): 52k examples generated by GPT-3.5 via self-instruct, lightly filtered
- 3Dolly v2 (Databricks): ~15k human written instruction/response pairs by Databricks employees
- 4OpenAssistant (LAION): community contributed, multi-turn, human curated conversations
- 5FLAN v2 (Google): millions of templated examples mapping classic NLP datasets to instruction format
Rank by how directly humans shaped each example: LIMA (hand-curated) > OpenAssistant > Dolly v2 > FLAN v2 (templated) > Alpaca (LLM-generated).
Imagine grading homework by how much care went into each answer, not how many answers there are. LIMA is a thousand essays each polished by an expert, so every example teaches a lot. OpenAssistant is real volunteers writing and rating conversations, still very human. Dolly is staff writing answers on the clock, good but rushed. FLAN reshapes old quiz datasets into one shape, so it is clean but repetitive and narrow per example. Alpaca had an older chatbot write its own homework with light checking, so each example is the cheapest and noisiest. More polish per item means more learning per item, even if the pile is smaller.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This ordering question looks like trivia about five datasets, but it is really probing one production instinct: do you rank training data by how much each example teaches, or by how many examples there are. The phrase quality per example is the whole game. It deliberately separates per-item signal from aggregate volume, and the five datasets are chosen so those two metrics point in nearly opposite directions.
The correct order, highest to lowest per-example quality, is LIMA, OpenAssistant, Dolly v2, FLAN v2, Alpaca. Notice that this is almost the reverse of a size ranking. LIMA has a thousand examples and tops the list. Alpaca has 52 times more and sits last. FLAN v2 has millions and lands in the lower half. If your mental model says more data is always better, this question is engineered to expose that.
The organizing principle is authorship provenance. As you move from expert hand-curation to scaled human writing to mechanical templating to LLM self-generation, the care invested in each example drops, and so does what the model extracts from it. This deep dive walks each rung of that ladder, explains the one genuinely tricky case (FLAN), and ends with the senior nuance that flips the whole ordering under a large training budget.
The ladder of authorship provenance
Every instruction dataset is produced by some process, and that process sets a ceiling on per-example quality. The ladder, from most to least careful, runs: expert hand-curation, then scaled human writing, then templating from existing benchmarks, then LLM self-generation.
The intuition is that a model learns the average quality of the distribution it is trained on. An example a researcher polished encodes more useful signal than one a model generated and a script lightly filtered. Curation removes the long tail of bad examples, which is exactly where fine-tuning damage comes from.
Map the five datasets onto the ladder and the order falls out almost mechanically:
- LIMA sits at the top rung, expert hand-curation.
- OpenAssistant is scaled human writing plus human rating.
- Dolly v2 is scaled human writing with lighter curation.
- FLAN v2 is mechanical templating from benchmarks.
- Alpaca is LLM self-generation with light filtering.
The only judgment call is where templating sits relative to fast human writing, which the FLAN section below resolves.
One caveat keeps this a heuristic rather than a rule. Provenance predicts the expected quality of an example, not its guaranteed quality. A heavily filtered synthetic set can outrank a sloppily collected human one. For these five specific datasets, though, the ladder lines up cleanly, which is why the question uses them.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Dataset | How produced | Per-example quality |
|---|---|---|
| LIMA | Hand-curated by researchers (1k items) | Highest, every example polished |
| OpenAssistant | Volunteers write and rate dialogues | High but contributor variance |
| Dolly v2 | Employees write ~15k pairs on the job | Solid human, light curation |
| FLAN v2 | Templated from NLP benchmarks (millions) | Clean but narrow and repetitive |
| Alpaca | GPT-3.5 self-instruct, light filtering | Lowest, noisy and shallow |
Real products, models, and research that use this idea.
- LIMA fine-tuned a 65B Llama base on just 1000 examples and matched far larger instruction sets in human preference tests, popularizing the curation-first thesis.
- OpenAssistant's human-rated conversation trees seeded the preference data behind several open chat models and remain a reference SFT corpus on Hugging Face.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you empirically rank two SFT datasets if you could only run a few fine-tunes?
Hold base model, hyperparameters, and example budget fixed, then compare on a shared eval. Subsample each set to equal size to isolate per-example quality from volume effects.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Conflating dataset size with quality. Alpaca has 52x more examples than LIMA, yet ranks last per example because LLM self-generation injects noise that hand curation removes.
60 second bullets to scan on the way to the call.
Quality-per-example versus total dataset size
The authorship ladder from expert curation to LLM generation
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.