Match instruction dataset to its source / construction method
Sort each dataset by who wrote the words and how: hand-curated (LIMA, Dolly), LLM-generated (Alpaca), templated benchmarks (FLAN), community multi-turn (OpenAssistant), scraped logs (ShareGPT).
Imagine six cookbooks. LIMA is 1000 recipes a chef perfected by hand to prove quality beats quantity. Dolly is recipes thousands of office workers wrote during a fun company drive. Alpaca had a robot churn out 52000 recipes cheaply from a few seed ideas. FLAN took thousands of old exam questions and reworded them all into one recipe format. OpenAssistant is a wiki where volunteers wrote conversations and voted on the best replies. ShareGPT is screenshots people posted of their chats with a famous robot, scraped into a pile. Knowing who cooked each one tells you why it tastes the way it does.
Detailed answer & concept explanation~7 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5 min: the three sourcing mechanisms, then walk each of the six datasets by author, scale, and method, then close on how provenance predicts the failure mode and the LIMA lesson.
| Dataset | Who wrote it | Scale | Mechanism |
|---|---|---|---|
| LIMA | Meta researchers | 1k | Hand-curated, quality over quantity |
| Alpaca | An LLM teacher | 52k | Self-instruct from seed set |
| Dolly v2 | Databricks employees | ~15k | Hand-written, categorised buckets |
| FLAN | Existing benchmarks | Very large | Templated conversion of supervised data |
| OpenAssistant | Volunteer community | Tens of thousands | Multi-turn dialogue with ranking |
| ShareGPT | ChatGPT users | Tens of thousands | Scraped conversation logs |
Real products, models, and research that use this idea.
- Stanford Alpaca kicked off the 2023 open-instruction wave by self-instructing 52k examples for roughly 500 USD, spawning Vicuna, Koala, and dozens of clones.
- Databricks shipped Dolly v2 with a fully commercial-use licence precisely because the 15k pairs were employee-written, not distilled from a closed model.
- Modern open models like Llama 4 and Qwen-class releases still use FLAN-style templated mixtures as one slice of their SFT data blend.
- LAION's OpenAssistant (OASST1) is still mined for both SFT pairs and DPO preference pairs thanks to its embedded human rankings.
- ShareGPT logs underpinned the original Vicuna, but later projects dropped them over the terms of service question around scraping ChatGPT.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy did Alpaca, despite 52k examples, often lose to LIMA's 1000 on alignment quality?
QWhat failure mode does a templated dataset like FLAN introduce that a conversational set avoids?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Mixing up Alpaca and LIMA. Both feel small and clever, but Alpaca is 52k cheap LLM-generated examples while LIMA is 1000 obsessively hand-curated ones with the opposite thesis.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.