Held Out Set

Spot the flaw: 'We fine-tuned on 10K examples and eval on a random 500 from the same dataset. Accuracy is 94%, so the model is ready for production.'

Design the post-FT eval suite for a customer-support fine-tune