Which quality controls are essential when using an LLM to generate eval test cases?

**Synthetic eval generation** means asking an LLM to write your test cases instead of hand-authoring them. It is attractive because it scales: one prompt can produce hundreds of questions across topics in minutes. But a raw generated set inherits the generator's habits, so you need quality controls. Four of the options are real controls. **Embedding-based dedup** removes near-duplicate questions that cluster around a few easy phrasings, so your set covers breadth, not the same idea fifty times. Adversarial seed prompts force the generator toward edge cases and failure modes it would otherwise dodge. Human spot-checking a sample catches factual errors and circular reasoning that automated signals miss, especially in reference answers. Cross-family validation runs the generated questions past a model from a different family to expose blind spots the generator baked in. The two wrong options are traps. Using the same model to both generate and evaluate the set creates an echo chamber: it rewards its own style and hides its own gaps. And generating 100,000 questions confuses size with quality. A stratified set of 200 good questions usually beats 100,000 degenerate ones.

**Synthetic eval generation** trades human authoring time for the generator's biases, so the quality controls all exist to claw back coverage and correctness. Dedup comes first. Generators cluster around a few easy phrasings. Embed each candidate, cluster by cosine similarity, and drop near-duplicates so the set spans the space. Adversarial seeding fixes coverage. Left alone, a generator writes safe, central questions. Seed prompts that demand edge cases and known failure modes push coverage into the long tail where models break. Human spot-checks are the backstop. Sample 50 to 100 items and have a domain expert read them. This catches fabricated facts and circular reference answers that pass every automated filter. Cross-family validation closes the loop. A generator's blind spots leak into its questions. Run the set past a different-family model to surface gaps and echo-chamber effects. The two distractors are classic traps. Using the eval target itself as the generator creates self-favoring alignment, not real signal. And a 100,000-item count is not a quality metric. Stratified breadth and verified correctness beat raw volume every time.