Match instruction dataset to its source / construction method
Drag each answer to line up with its matching prompt
LIMA
Community sourced multi-turn conversations with human review and ranking, released by LAION.
Alpaca
~15k pairs hand written by Databricks employees across categorised task buckets.
Dolly v2
52k self-instruct examples generated by an LLM (text-davinci-003) from a small seed set.
FLAN
Scraped logs of user conversations with ChatGPT, popular as a base for early open source instruction tuning.
OpenAssistant
Templated conversion of many classic supervised NLP datasets into instruction format at large scale.
ShareGPT
1000 hand curated examples from Meta, designed to show that less but better beats more but noisy.
Sort each dataset by who wrote the words and how: hand-curated (LIMA, Dolly), LLM-generated (Alpaca), templated benchmarks (FLAN), community multi-turn (OpenAssistant), scraped logs (ShareGPT).
Imagine six cookbooks. LIMA is 1000 recipes a chef perfected by hand to prove quality beats quantity. Dolly is recipes thousands of office workers wrote during a fun company drive. Alpaca had a robot churn out 52000 recipes cheaply from a few seed ideas. FLAN took thousands of old exam questions and reworded them all into one recipe format. OpenAssistant is a wiki where volunteers wrote conversations and voted on the best replies. ShareGPT is screenshots people posted of their chats with a famous robot, scraped into a pile. Knowing who cooked each one tells you why it tastes the way it does.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Every instruction dataset is defined less by the topics inside it than by HOW it was constructed. The construction method predicts the dataset's strengths, its biases, and its failure modes.
All six fit a single grid with three columns: who authored the examples, at what scale, and by what mechanism. The author can be a human, an LLM, or a pile of pre-existing data. The mechanism follows from that: hand curation, self-instruct generation, templated conversion, or scraping.
Curated demonstrations beat raw volume, which is why LIMA exists as a deliberate counterpoint to Alpaca.
The three sourcing mechanisms
Every instruction dataset is built by one of three mechanisms. The mechanism is not a cosmetic label. It determines the cost per example, the upper bound on scale, the diversity of phrasing, and the legal status of the resulting weights.
The first is human authoring. People sit down and write (instruction, response) pairs by hand. This is the most expensive route and the highest quality per example, but it caps out at thousands rather than millions of examples. The signal is dense because a human writer brings judgement to every line. Yet coverage is narrow, because no team can hand-write a million prompts. LIMA and Dolly v2 live here.
The second is LLM generation. You give a teacher model a small seed set and let it synthesise thousands of new examples. Self-instruct is the usual method. Seed prompts go in, the teacher expands them into new instruction-response pairs, and a light filter removes near-duplicates. It collapses cost, but the output inherits the teacher's errors, its style, its refusal habits, and a licence that may forbid commercial use. Alpaca is the textbook case.
The third is reuse of existing data, in two flavours. Templating converts pre-existing supervised benchmarks into instruction phrasing at scale, which is the FLAN approach. Scraping harvests conversations that already happened, which is the ShareGPT approach. OpenAssistant is a fourth, crowd-sourced flavour where a volunteer community both writes the conversations and ranks competing responses. That makes it richer than a plain pile of pairs.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Dataset | Who wrote it | Scale | Mechanism |
|---|---|---|---|
| LIMA | Meta researchers | 1k | Hand-curated, quality over quantity |
| Alpaca | An LLM teacher | 52k | Self-instruct from seed set |
| Dolly v2 | Databricks employees | ~15k | Hand-written, categorised buckets |
| FLAN | Existing benchmarks | Very large | Templated conversion of supervised data |
| OpenAssistant | Volunteer community | Tens of thousands | Multi-turn dialogue with ranking |
| ShareGPT | ChatGPT users | Tens of thousands | Scraped conversation logs |
Real products, models, and research that use this idea.
- Stanford Alpaca kicked off the 2023 open-instruction wave by self-instructing 52k examples for roughly 500 USD, spawning Vicuna, Koala, and dozens of clones.
- Databricks shipped Dolly v2 with a fully commercial-use licence because the 15k pairs were employee-written, not distilled from a closed model.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy did Alpaca, despite 52k examples, often lose to LIMA's 1000 on alignment quality?
Frame around the self-instruct pipeline inheriting the teacher's errors and low diversity, versus hand curation maximising per-example signal. Quality density beats raw count once the model already has capability from pretraining.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Mixing up Alpaca and LIMA. Both feel small and clever, but Alpaca is 52k cheap LLM-generated examples while LIMA is 1000 obsessively hand-curated ones with the opposite thesis.
60 second bullets to scan on the way to the call.
Three sourcing mechanisms: human-written, LLM-generated, templated or scraped
LIMA's less is more thesis and its 1000 example scale
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.