Design a golden eval set for a production RAG system serving a legal research product. Explain how you build it, how big it should be, what each example contains, and what you measure with it.

A golden eval set is a fixed list of test queries with known good answers that you run every time the system changes. For a legal research RAG product, design it in three parts. First, source the queries from multiple streams: real user queries from production logs, SME-written questions from a panel of lawyers, and adversarial cases like multi-hop questions or queries with no answer in the corpus. Each stream catches different failures. Second, keep it small but high quality. Start with 200-500 examples, not 5,000. Each example needs a gold answer, the gold supporting chunks, a 'has_answer' flag, and an acceptable answer rubric. SME labeling is the real cost. Third, score four axes: retrieval recall (did the right chunks come back), faithfulness (does the answer follow from the chunks), answer correctness (does it match the gold answer's content), and refusal correctness (did the system refuse when it should). Use LLM-as-judge for the qualitative axes. Gate every release on the eval; no regression beyond a small budget is allowed.

**Three sourcing streams** combined give you coverage no single stream provides. Real production queries (last 30-90 days, stratified by user type and query length) capture the actual workload. SME-written questions from a 5-10 person panel of legal researchers catch systematic blind spots that real queries miss. Adversarial cases (multi-hop, no answer in corpus, pre-cutoff case law) probe known failure modes. One stream alone is insufficient. **Start at 200-500 examples.** The bottleneck is SME labeling cost, not query count. Each example needs five fields: the gold answer text, the list of gold supporting chunk IDs, a has_answer boolean, an acceptable answer rubric, and tags. The chunk IDs make retrieval recall measurable. Legal answers have many valid phrasings, so the rubric matters. Tags cover multi-hop, refusal required, citation required, jurisdiction. Grow incrementally as new failure modes surface. **Score four axes.** Retrieval recall@k checks whether at least one gold chunk appears in the top-k. Faithfulness uses LLM-as-judge (Claude Opus 4.7 or GPT-5.5) to verify every claim follows from retrieved chunks. Answer correctness uses LLM-as-judge against the rubric; exact match is meaningless for prose. Refusal correctness checks has_answer=false cases. **Gate every release.** No more than 2-point regression on any axis vs production, no regression on the adversarial subset. Hold out a quarterly only subset to detect overfit to the dev set. In legal, a single hallucinated citation is a customer trust catastrophe; the eval is non-negotiable.