What's the difference between iterating on prompts by 'feel' (vibes-driven) versus by eval-driven discipline and why does it matter in production?

**Vibes-driven iteration** means changing a prompt, looking at the output on a few examples you have in front of you, and shipping if the output looks better. It is fast and feels productive, but the test set is whatever happened to be open in your editor that day. The failure mode is silent regression. The change you made may have improved the three examples you tested while quietly degrading twenty cases you did not look at. You only find out when a user reports the regression in production or when a downstream metric starts moving the wrong way. **Eval-driven iteration** flips the test set from ad hoc to held-out. Build a golden set of 100 to 500 representative inputs with expected outputs or assertion criteria. Pick concrete metrics (faithfulness, format compliance, refusal rate, structural conformance). Every prompt change runs against the set as a regression gate. The change ships only if the metrics hold or improve. The distinction matters most in production. Vibes is fine when you are exploring; eval is what you need to keep quality stable as the prompt evolves across many edits and many hands.

The difference is whether prompt iteration is grounded in a held-out, measurable evaluation. Vibes-driven iteration tweaks the prompt, hand-checks a few examples, declares the result better or worse, and ships. The implicit test set is the small number of cases you happened to have open. The implicit metric is whether the output looks right to you. Both are private to the iteration and impossible to share, so quality cannot be tracked over time and silent regressions on cases you did not test go undetected. Eval-driven iteration grounds every change in a held-out golden set, typically 100 to 500 input-output pairs covering representative and edge cases. Each output is scored by concrete metrics (faithfulness against retrieved context, format compliance against a schema, refusal rate on blocked categories, structural conformance to a template). Every prompt change runs against the suite as a regression gate. The change ships only if the metrics hold or improve. The production-relevance gap is that vibes scale to one prompt iterated by one engineer. Production prompts get edited by many hands across many sprints. Without a regression gate, the prompt slowly degrades on cases nobody is currently testing. Eval-driven discipline turns prompt iteration into a measurable engineering practice. You can see exactly which change moved which metric, and you catch silent drift at PR time rather than in user reports. The right framing is not eval versus vibes. Vibes is fine for exploration. Eval is required once the prompt is in production and any regression matters.