Eval Dataset Construction

An 18 month old golden eval set, departed annotators, three model updates, shifted production traffic. Design the lifecycle management process.

In classical ML the data splits into train/val/test. For LLM evaluation, which split matters most and why?

How do you use an LLM to generate eval test cases, and what quality controls prevent degenerate sets?

Which quality controls are essential when using an LLM to generate eval test cases?